NL-2-SQL Lab

Enterprise warehouses have 100+ tables across 8+ schemas. A naive LLM guesses table names from training data or picks the first semantic match. A wrong table means a confidently wrong answer — with no error.

"Revenue" exists in 4 tables: advisory_fees, transactions, revenue_adjustments, and a GL summary. They measure different things at different grains. Without domain context, an LLM picks randomly — and is wrong most of the time.

AUM should be a monthly daily average — not a month-end snapshot and not a sum. Using the wrong grain produces trend lines that are distorted by month-end market volatility, making the answer look right but be wrong.

Critical tables have RLS policies enforced at the database layer. A naive query that joins on a non-RLS column either silently returns partial data or fails with a cryptic permission error — both worse than a clean block.

A JOIN to core.clients exposes PII (name, email, SSN, tax_id) even when those columns are not in the SELECT list — the query execution plan still reads the PII block. Standard data policies require column-level masking at the source.

A single unguarded query on a 890M-row partitioned table without a date filter can run for hours, consume terabytes of I/O, and degrade production databases for all other users. LLMs don't know table sizes or partition schemes.

"Why is revenue down while AUM is up?" cannot be answered with a single SQL query. The answer requires: revenue trend, AUM trend, fee rate computation, and product mix — in that dependency order. A single query answers nothing.

"Last quarter", "Q2", "recent", "this year", "year to date" are all ambiguous. LLMs resolve them differently across runs, producing non-deterministic SQL. The same question asked twice can return different time ranges.

LLMs confidently reference columns that don't exist, invent table aliases from training data, or use column names from different systems. The query fails at runtime — or worse, silently matches a wrong column with a similar name.

Joining a fee table to a holdings table without pre-aggregation multiplies rows: each fee row joins to N holding rows, inflating SUM(fee) by factor N. The result is numerically plausible but completely wrong. No error is raised.

Raw query results — a table of numbers — are not an answer. A business user who asked "why is revenue down?" does not benefit from seeing [(Jan, 2.34M), (Feb, 2.18M)…]. The last mile interpretation is still entirely manual.

When 12% of clients have no CRM records, or a materialized view is 26 hours stale, the query still returns a result. A naive system presents it as a complete answer. In risk detection, a missing client looks "safe" — the opposite of reality.