Kenroy
Green

My data engineering knowledge base started as straight RAG. Every question followed the same path: embed the query, pull the nearest chunks from the vector store, hand them to the model, return an answer. It worked for lookups. It fell apart on everything else.

A fixed retrieval path retrieves whether it should or not. Ask it to write a SQL query and it would still go digging through documentation chunks first. Ask it something current and it had nothing, because the only knowledge it could reach was whatever I had indexed. The pipeline was making the decision, not the model.

So I rebuilt it as an agent using OpenAI function calling. Now the model decides whether a tool is needed at all, which one, and in what order. It has three: knowledge base retrieval over Pinecone and LlamaIndex, SQL generation that is aware of the target dialect, and web search through Tavily for anything outside the index. A simple question gets answered directly. A harder one gets the right tool, and the model can chain several across multiple reasoning rounds before it responds.

It also keeps conversation memory across turns, so follow-up questions build on what came before instead of starting cold.

The part I care about most is failure handling. When a tool errors, I return the error back to the model rather than dropping the request. It reads what went wrong and tries another route. A failed SQL call or an empty search does not end the conversation, it just becomes the next thing the model reasons about. That one decision made the whole system feel reliable instead of brittle.

Same models, same vector store, same data. The difference is who is deciding. Handing that choice to the model instead of hard-coding it in a pipeline is the line between a retrieval script and something that can actually work a problem.

Live demo →   View code →

I've spent 15 years building the platforms. SQL Server, Azure Data Factory, Databricks, dbt, Power BI. ETL pipelines, dimensional models, Bronze/Silver/Gold architectures. The stuff that turns raw source data into something trustworthy.

I just completed the Associate AI Engineer track on DataCamp. 13 courses, 2 projects, 40 hours. PyTorch, Hugging Face, LLMs, MLOps, the lot. Not because I'm switching careers, but because the best AI models in the world are useless if the data feeding them is inconsistent, undocumented, or arriving late.

I've seen it across every industry I've worked in. Reinsurance, FMCG, media, private equity. The bottleneck is never the model. It's the data platform underneath it.

The companies that will get real value from AI aren't the ones hiring prompt engineers. They're the ones whose data foundations are solid enough to support it. Clean pipelines, proper lineage, tested transformations, governed access.

That's the work I do. And understanding what sits on top of it makes me better at it.

A stored procedure at a global reinsurer was eating 2 to 3 hours of daily runtime. Nobody had touched the code.

Same logic. Same proc. Same everything. It just got slower every year while the data quietly grew from 4 million rows to 86.8 million. Nobody noticed until it became a problem everyone worked around.

I pulled the actual execution plan and found two problems.

First, two missing indexes on the primary join column across the two largest tables (86.8M and 7.7M rows). Every execution was doing full clustered index scans on both. I created covering nonclustered indexes with INCLUDE columns for every field the procs actually read. The optimiser switched to index seeks immediately. Runtime dropped from 15–16 minutes to 5 minutes. 67% improvement from two indexes.

Second, an architectural issue the indexes alone couldn't fix. The original proc joined all 7 tables at the finest grain before aggregating, producing a Sort operation across 93 million rows that ate 59-64% of total execution cost. I rewrote the proc to pre-aggregate into a temp table at a coarser grain first, collapsing 86.8 million rows down to 7.8 million before any joins happened. The Sort dropped from 93 million rows to 7.8 million.

Both new versions were validated row-for-row against the originals. Identical results. The original procs were left untouched until the new ones passed Dev and UAT side by side.

Performance problems at scale are rarely about bad query logic. They're about what the database has to physically do to satisfy it. The proc was fine when the data was small. The data grew. The infrastructure around it didn't. A proper execution plan analysis found the real causes in hours, not weeks.

If you've got a stored procedure that "just got slow", look at the execution plan before you rewrite anything.