Skip to content
All projects
BACKEND/AIFintech · speech analytics

Banking Call Intelligence

Hybrid rules + AI analysis of financial calls.

Started
Feb 2026
Stack
FastAPI · Whisper · React · Python

01 · Problem

The problem

Banks handle thousands of calls a day, and the intent, urgency and risk in them stay locked in audio. Manual review doesn't scale, and a pure LLM pipeline can hallucinate, which a regulated bank can't accept.

  • Every decision needs to be explainable and auditable.
  • Account and card numbers must not be stored in full.
  • If the AI layer is unavailable or answers badly, the system still has to give an answer.

02 · Approach

How it works

  1. 01Speech to textOpenAI Whisper (the base model) transcribes each uploaded call. The request returns a call ID once the call has been analysed, and the dashboard fetches the result by that ID.
  2. 02Deterministic first passKeyword rules checked in a fixed order classify the intent; keyword sentiment sets priority and risk (a negative call is High on both); account and card numbers are masked to their last four digits.
  3. 03AI as a reviewerA Backboard assistant running GPT-4o reads the transcript with the rule output and returns a verified intent, a verified priority and its reasoning, which become the final decision.
  4. 04A fallback for every failureIf the AI is not configured, returns nothing, returns something that is not valid JSON, or errors, the rule-based result stands, and the rule result is stored next to the AI's for every call.
  5. 05Hardened APIFastAPI with API-key authentication (an X-API-Key header) and per-IP rate limiting with SlowAPI, 10 uploads and 30 result checks a minute, behind a React dashboard with risk alerts, a queue and per-client and per-category views.

03 · Architecture

Architecture

Rules decide first, AI reviews, and the rules stand whenever the AI fails.

CALL AUDIOReact dashboardFASTAPIAPI key · rate-limitedWHISPERspeech to textRULE ENGINEintent · risk · maskingAI VERIFIERBackboardDECISIONmerge or fall backRESULTSpolled by the UI
  1. 01
    A call is uploaded

    From the React dashboard.

  2. 02
    FastAPI takes it

    Checked against the API key and rate-limited with SlowAPI.

  3. 03
    Whisper transcribes

    The audio becomes a full transcript.

  4. 04
    Rules decide first

    Intent, sentiment, priority and risk from keywords, with account and card numbers masked.

  5. 05
    AI double-checks

    The verifier reviews the transcript and the rule output and explains its view.

  6. 06
    Merge, or fall back

    The AI's verified answer is used; if the AI fails, the rule-based decision stands (dashed line).

  7. 07
    Results reach the dashboard

    Risk alerts, the queue and per-client and per-category views update as results land.

04 · Results

Design guarantees

There's no labelled benchmark for this project yet, so these are guarantees built into the design rather than measured scores.

decision layers: rules, AI review and fallback
3
a rule-based answer stored beside every result
Always
account and card numbers, to their last four digits
Masked
auth, plus per-IP rate limiting
API key
Next case studyIoT Fire DetectionMulti-sensor fire detection with a digital twin.