Cookbook
Executed, end-to-end notebooks: from your first database to event-driven backtesting, prediction markets and performance analytics. Every recipe runs top to bottom against real or deterministic synthetic market data.
Fundamentals#
01Quickstart: your first h5i-db market databaseh5i-db is an embedded, versioned time-series database for quant workloads. There is no server to run.02Designing market data schemasA table in h5i-db is an Arrow schema plus a time column, persisted as immutable, time-sorted Parquet segments under versioned manifests.03Ingestion patterns: five sources, one tableNo desk gets its data from one place. The tick feed hands you Arrow batches, research notebooks live in pandas or polars, vendors drop Parquet, and some…04A SQL tour for quantsh5i-db's query layer is Apache DataFusion: full SQL with joins, CTEs and window functions.05Time travel and versioning: which version did my backtest see?Every write to an h5i-db table is an atomic commit that produces a new immutable version. That covers append, write, delete and restore.06Previewable mutations: fix bad ticks without fearing the delete keyDeleting or rewriting rows in a shared tick store is the scariest routine operation on a quant desk.07Streaming appends and tail(): a live feed on a versioned storeAn h5i-db table with an append-only history doubles as a message log. Every append is one commit, commits are strictly ordered, and a reader that…08Maintenance: verify, compact, vacuumA versioned database makes an unusual bargain. It never overwrites data, so it accumulates manifests, segments and history. That is the feature.09The DataFrame builder: queries as Python objectsdb.table(...) starts a lazy query that you assemble with method calls instead of a SQL string. Nothing runs until a terminal call such as .collect().
Market data engineering#
01OHLCV bars from tick dataRolling ticks into bars is where most tick pipelines pick up their first bug: a bucket boundary off by one, a close taken in file order instead of event…02VWAP, TWAP and execution benchmarksExecution desks live and die by benchmark arithmetic: interval VWAP, TWAP, arrival price, slippage in basis points.03Regular grids for irregular markets: gapfill and resampleIlliquid names do not trade every minute. Almost everything downstream wants a regular time grid anyway: covariance matrices, joins against liquid…04ASOF joins: trades vs quotes, signing and spreadsAttaching the prevailing quote to every trade is the microstructure join. It powers trade signing, effective and realized spread measurement, TCA and…05Corporate actions: split adjustment without losing the tapeA stock split rewrites history. Every price before the effective date must be scaled, or every return computed across it is garbage.06Point-in-time fundamentals: killing lookahead bias with ASOF joinsFundamentals have two timestamps. periodend is the fiscal date the numbers describe.07Tick data cleaning: find it, preview the fix, keep the audit trailRaw vendor tick files arrive dirty. Fat-finger prints 10x off, zero prices from feed glitches, duplicated blocks from replayed packets, after-hours junk.08NBBO consolidation: best bid/offer across fragmented venuesUS equities trade on a dozen or more venues, each publishing its own top of book.
Alpha research#
01Cross-sectional momentum: an honest monthly backtestThe classic 12-1 momentum factor, end to end on real prices. ), and the signal table itself is versioned and snapshotted.02Pairs trading with a version-pinned data spineA cointegration pair strategy on real prices. We scan candidate pairs with an Engle-Granger test, build a rolling hedge ratio, compute the spread z-score…03EWMA volatility and vol-targeted position sizingRiskMetrics-style EWMA volatility is the workhorse conditional-vol estimate on every risk desk, and h5i-db ships it as a native SQL window function…04Realized volatility from ticks: signature plots, jumps, overnight riskRealized variance, the sum of squared intraday returns, is the standard nonparametric vol estimate.05Building a point-in-time factor libraryEquity factors die by lookahead. A B/P ratio computed with a book value the market had not seen yet will backtest beautifully and trade terribly.06Event studies: CARs with ASOF-aligned announcement datesThe classic event-study pipeline computes market-model abnormal returns and cumulative abnormal returns around announcements.07Order flow imbalance: does signed volume predict returns?Order flow imbalance is the excess of buyer-initiated over seller-initiated volume, and it is the workhorse microstructure signal.08Intraday seasonality: volume U-shape, volatility smile, spread decayAlmost every execution and alpha model conditions on time of day. Volume concentrates at the open and close, volatility peaks in the first hour, spreads…09Lead-lag discovery: cross-correlations on irregular ticksWho moves first? Lead-lag analysis pairs related instruments: index against futures, ADR against home listing, correlated FX crosses.10Portfolio rebalancing with versioned holdingsA portfolio book is the canonical versioned dataset. " become version queries rather than spreadsheet archaeology.11Retrieval-augmented forecasting: historical analogs as a knowledge base"What happened the last twenty times the tape looked like this?" Analog forecasting is one of the oldest ideas in the business. Retrieval-augmented…12Trend following: each asset against its own pastRecipe 02/01 ranks assets against each other and buys the winners. This one never compares two assets at all: each is measured against its own history…13Short-horizon mean reversion, and the edge it would have neededOver a year, winners keep winning. Over a few days the textbook says they hand it back: a name that fell hard against its peers bounces, because part of…
Risk & production#
01VaR and Expected Shortfall with an auditable risk tableA risk number nobody can reproduce is a liability.02Reproducible backtests: pin the data, not just the codeEvery quant team has lived this incident. A backtest from March cannot be reproduced in July.03EOD snapshots and the audit trail regulators actually ask forThe question that arrives eighteen months later is never "what is the price now". ".04Data-quality gates: staging, policy, and previewable remediationThe worst place to discover a broken vendor file is in the P&L meeting.05A crash-safe paper-trading loop with full order attributionThe hard part of a live loop is not the strategy. It is answering, a week later, "why did we send that order?".06Multi-writer coordination: optimistic locking, conflicts, and retriesAn h5i-db database is a directory, and nothing stops two processes from opening it at the same time: a feed handler and a corrections job, or two…07Options: implied-vol surfaces as versioned marksA vol desk's surface is not one object. It is a sequence of marks: EOD snapshots, intraday re-marks, corrections.08FX and crypto: 24/7 data without an exchange sessionEquity tooling leans on the session. The exchange defines "the day", the open and the close.09Fixed income: versioned curve marks, restatements, carry & rolldownA rates desk's core dataset is small but unforgiving. One par curve per mark date, and every number on it feeds risk, P&L and client marks.10Performance tuning: pruning, projection, commit granularity, cachesh5i-db stores each table as immutable, time-sorted Parquet segments under a versioned manifest.11arrival-delta: which of last night's backtests actually held up?A research agent runs forty backtests overnight. So does a parameter sweep, or a junior with a for-loop. In the morning there are forty Sharpes.
Event-driven backtesting#
01Your first event-driven backtestThe backtests in section 02 are vectorized. Compute a signal for every date, multiply it by the return that followed, and sum.02Stress-test execution assumptionsEvery backtest contains execution assumptions, and most of them are never written down. Fills happen at the price you asked for.03Operate reproducible backtestsA backtest is a claim about the past, and it is worth what the evidence behind it is worth.04A production data contract for Kaggle Polymarket L2A public dataset is not a research input. It is a pile of files carrying whatever timestamps, units and duplicate rows the recorder happened to produce…05Causal signal replay on real Polymarket booksSynthetic data proves the plumbing works. It cannot tell you whether a strategy works, because what you find in it is the structure the generator put…06Order lifecycle and account riskMost backtests model an order as a single event. It is sent and it fills. Real orders have a life.07Path-dependent Python strategiesSome strategies cannot be written as a table of order intent. A rule that enters only once the previous position is confirmed closed, or that waits…08From a vectorized equity backtest to an event-driven oneSection 02 backtests a monthly momentum rule by multiplying a signal by the return that followed. Section 04 has so far replayed prediction markets.09Market making: inventory, latency, and being run overEvery other recipe in this section takes liquidity. A market maker supplies it, and the job is different in kind. There is no forecast.10Execution algorithms and the cost of not finishingRecipe 01/02 measures VWAP and TWAP as benchmarks. This one trades to them. The order is 20,000 shares to buy in a name that shows a few hundred at the…11Calibrating costs from your own fillsRecipe 04/02 varies fees, slippage and latency to see whether a conclusion survives them.12Searching a strategy space without fooling yourselfA backtest that has been run once is a measurement. A backtest that has been run four hundred times and reported once is a selection, and the number on…13Replaying an account's ledgerEvery other recipe in this section asks what a strategy would have done. This one asks the strictest question a backtester can be asked: here are the…
Prediction markets#
01Binary parity and the fee curveA binary event contract has an arithmetic identity: YES and NO settle to exactly 1.00 between them. When both offers sum to less than 1.00 you can buy…02Is the market's probability any good?A prediction market quotes a probability, so it can be scored the way any forecast is scored.03Trading the favorite-longshot biasThe oldest documented anomaly in event contracts is that longshots are overpriced and favorites underpriced.04What your data can and cannot tell you about executionPrediction-market research usually runs on periodic book snapshots, because that is what the public APIs give you.05Two ways a prediction-market backtest lies to youThe first is settlement. A replay that stops before a market resolves still holds a position, and marking it to the eventual winner books money nobody…06From a vendor mirror to canonical tablesMost prediction-market research starts with a directory of vendor Parquet already on disk: hourly full-feed archives, or per-day channel files.07The whole loop, onceVendor files to a decision, with nothing skipped: ingest, pin, build a strategy from the pack, search it walk-forward with the holdout spent on a…08The whole loop on real Polymarket booksRecipe 05/07 runs the pipeline on a synthetic panel, which proves the plumbing and nothing about the world.
Performance analytics#
01Tearsheets and performance statisticsEvery recipe so far ends with a P&L number and a chart drawn by hand. That is fine for one strategy and hopeless for twenty, and it is how two people end…02Factor panels: IC, quantiles, turnoverRecipe 02/05 builds a factor and scores it with pandas and scipy. That is the right way to learn what an information coefficient is, and the wrong way to…03Cross-validation that a time series survivesA backtest measures what happened. Cross-validation is supposed to answer a harder question: was the parameter that won chosen for a reason, or was it…04Sweeps, verification, and what a restatement did to your alphaThree questions arrive in order at the end of a piece of research. What did the parameter grid say? Can somebody else reproduce the number?