Building 'Finn': How We Engineered an Intelligent Market Scanner

A behind-the-scenes look at the architecture of our proprietary tool that processes 25k+ daily datapoints to find arbitrage opportunities.

At Datajul, we don't just build tools for clients; we build them for ourselves. "Finn" is our internal market scanner, designed to identify arbitrage opportunities and fundamental anomalies across global markets.

The Architecture

Finn is built on a robust stack designed for speed and reliability:

• Data Ingestion: Python scripts running on AWS Lambda fetch real-time price data from the FMP Enterprise API.
• Orchestration: n8n workflows manage the data pipeline, handling error retries and dependency management.
• Storage: A PostgreSQL database stores historical data for backtesting strategies.
• Analysis: Pandas and NumPy perform vectorised calculations to identify price discrepancies in milliseconds.

Why Build Internal Tools?

Building Finn allowed us to test our data engineering capabilities to the limit. Processing 25,000+ datapoints daily requires robust error handling and efficient code. The lessons we learned building Finn directly translate to better, more resilient solutions for our clients.

Results

Since its deployment, Finn has identified numerous market inefficiencies that would have been impossible to spot manually. It serves as a testament to the power of automated financial data engineering.