Can data agents do real analysis?

Can an AI agent do the work of a junior data analyst? Shreya Shankar, an incoming database professor at Carnegie Mellon University built a benchmark to find out. Her benchmark uses messy conditions: multiple databases, tribal knowledge, and unstructured text. In her tests, the best agents solved about 61% of queries on the first try. Tristan and Shreya get into why agents tend to pull everything into pandas instead of pushing computation down to the source, and why a good semantic layer makes such a big difference. They also cover why plugging agents into raw data goes badly while gold-layer tables with a semantic layer work much better. And they ask whether agents need their own version of the visual shortcuts humans use to spot what's interesting in data. They also cover evals. Shreya explains "criteria drift," the finding from her paper Who Validates the Validators? that people don't know what they want from an LLM judge until they see the outputs. She also explains why she wrote In Defense of AI Evals, and what practitioners can do today when exploratory data analysis doesn't map cleanly onto agent outputs. Finally, she previews her research at CMU on making AI SQL cheaper, including a custom inference engine for AI-powered queries. The Analytics Engineering Podcast is sponsored by Fivetran + dbt Labs.