Automotive Cybersecurity RAG System
Built to eliminate the inefficiency of cybersecurity engineers manually searching through ISO/SAE 21434, UN R155, and internal TARA tables for every query. The TARA automation tool (tAIRA) calls this system at each analysis step to pull in grounding context.
Role: Solely designed and implemented the entire RAG pipeline — from embedding model selection to retrieval strategy, LLM answer generation, and API integration. The core challenge was finding the best architecture under two constraints: 'identifiers must be found exactly' and 'sensitive data must never leave the company network.'
A query is embedded as dense+sparse via BGE-M3, retrieved via FAISS, scores combined at 0.7/0.3, reranked by a cross-encoder to the top 5 chunks, and passed as grounding to Ollama (qwen2.5:32b) for Korean answer generation. Documents are chunked and embedded offline with incremental indexing.
Design Rationale
I initially assumed dense search alone would suffice, but queries like 'What is threat M013-1?' returned irrelevant results — semantic embeddings can't distinguish meaningless identifier codes. So I switched to a hybrid approach, and BGE-M3 conveniently produces both dense and sparse vectors in a single encoding, keeping the pipeline simple. The cross-encoder dramatically improved relevance but was too slow to apply to all results, so I limited it to the top 20 — a tradeoff between accuracy and latency. For the LLM, GPT-4 would have been better, but TARA data is confidential and cannot leave the company network, so I accepted some quality loss and deployed Ollama locally. The GPU decided the model size: the ceiling for the L4's 24GB in a g6.2xlarge was qwen2.5:32b at Q4_K_M (~20GB), so anything larger was never on the table.
Running separate dense and sparse models would have complicated the pipeline. BGE-M3 produces both vectors in a single pass, keeping the architecture simple.
I considered Milvus and Weaviate, but with only a few thousand documents, spinning up a dedicated vector DB server felt excessive. File-based FAISS was sufficient and simpler to deploy.
First-stage retrieval ranking wasn't satisfactory. The cross-encoder evaluates query-chunk pairs together for much more accurate relevance, but it's too slow for all results — so I limited it to the top 20.
GPT-4 gave better answers, but TARA data cannot leave the company network, so the field narrowed to models we could host ourselves — and qwen2.5 had the best Korean performance among them. The GPU decided the size: the NVIDIA L4 in a g6.2xlarge has 24GB of VRAM, and qwen2.5:32b at Q4_K_M lands around 20GB, effectively the ceiling for a single card. Dropping to 14b would have left more headroom, but the quality gap was visible when summarizing long security-standard passages, so I accepted the constraint that little VRAM remained for context. That is why reranking down to the top 5 chunks was a requirement rather than an optimization.
tAIRA is Python-based, so using the same language for integration was natural. Since all inference calls are synchronous blocking, I needed an async framework to delegate them to a thread pool via asyncio.to_thread.
Public standard documents are baked into the image, while sensitive TARA data and indexes are mounted from host volumes. This allows image-only updates for deployment without risking data leakage.
- Issue
- Queries like 'Explain threat M013-1' returned irrelevant results instead of the exact matching item.
- Analysis
- Dense embeddings can't distinguish meaningless codes like 'M013-1' from general words like 'automotive' or 'security.' The reranker made it worse by splitting identifiers into subwords, actually weakening exact matching.
- Solution
- Combined dense (0.7) + sparse (0.3) scores to reinforce exact token matching, and added a branch: when an ID pattern (M013-1, ISO 15.4, etc.) is detected, bypass the reranker and boost sparse exact matches instead. The key insight was abandoning the assumption that all queries should go through the same pipeline.
- Result
- Identifier queries now reliably surface the correct item at the top, while general queries still benefit from the reranker for relevance.
- Issue
- With 2+ concurrent queries, later requests stalled until the first finished — embedding, FAISS, reranker, and Ollama calls are all synchronous blocking.
- Analysis
- FastAPI is async, but inference libraries are synchronous. Calling them directly inside async functions blocks the entire event loop.
- Solution
- Delegated all blocking calls to a thread pool via asyncio.to_thread. Also designed tAIRA integration to gracefully degrade with empty context on RAG failure — RAG is an auxiliary tool and should never block tAIRA's core analysis flow.
- Result
- Event-loop blocking under concurrent queries disappeared, and tAIRA's TARA analysis continues uninterrupted even when RAG fails.
The company is an automotive cybersecurity consultancy, not a software organization, so I built this system alone from scoping through deployment. Writing a RAG pipeline with no one to review the code and no one to argue the design with was the real constraint. I worked with AI as a pair programmer and filled the missing reviewer's seat with test code instead.
I pinned down tests for how retrieval shifts when the chunk size changes, whether identifier queries actually route into the hybrid branch, and whether concurrent requests ever cross responses — and once implementation was handed off, I ran the tests first, every time. When nobody else is reading your code, what you have already verified is the only reason to trust it. What I regret is that the verification stopped at behavior. I never built retrieval-quality metrics like recall@k, so every parameter change still came down to a judgment call about whether it was an improvement or a regression.
At the same time, the limits of what AI could do for me were obvious. Why dense embeddings alone can't retrieve an identifier like M013-1, why the cross-encoder had to be capped at the top 20 rather than applied to everything — those calls were only available to me because I understood the problem. AI turned my decisions into code quickly; it never made the decisions. Building at this scale alone was possible because of AI, but what made it actually run was leaving a reason and a test behind every choice.