Real-Time Bitcoin Anomaly Detection

The Problem

Cryptocurrency markets move fast and generate huge volumes of tick data — flagging anomalous price or volume behavior only matters if it happens in near real time. The goal was a pipeline that could ingest and process thousands of records a day with sub-second detection latency, without relying on brittle manual thresholds.

Approach

1. 01Ingested raw data through a Kinesis data stream, with a Python processing layer computing derived features and pushing them onto a second, processed Kinesis stream.
2. 02Ran a Lambda function on the processed stream that scores each record with a trained Isolation Forest model, firing an SNS alert immediately whenever an anomaly is flagged.
3. 03Delivered both raw and processed data to S3 via separate Firehose streams, keeping a full historical record independent of the real-time alerting path.
4. 04Queried the S3 data with Athena and built a QuickSight dashboard for real-time and historical visualization.

Visuals

Results

Reflection

Moving from a statistical threshold baseline to a learned Isolation Forest model was the single biggest improvement — thresholds are brittle and need constant re-tuning as market volatility shifts, while the model adapts to the shape of 'normal' data automatically.