LLMs appear to vindicate the "bitter lesson": AI researchers succeed less by incorporating domain knowledge and more by scaling data and compute. Taken as prescription, “all you need is scale”. Indeed, practitioners in many domains have moved from traditional ML methods with domain-driven feature engineering to foundation models and other deep learning approaches. However, theoretical and empirical facts complicate this picture. Data alone, however massive, cannot identify the function we want to learn: identical training pipelines can agree on held-out test data yet diverge sharply outside that distribution. Despite this, today's deep learning systems generalize impressively well. If training data alone can't determine which of many possible internal models is "correct," where does this success come from? One answer: we never stopped imposing structure; we only made it implicit, built into architecture and learning algorithms rather than features. Tabular data shows what happens without it. No architecture has yet captured its implicit structure the way convolutions capture images or attention captures sequences, so industry still relies on traditional methods and feature engineering other domains moved past. The opportunity is to make that structural choice deliberate, and nowhere is it more available than in tabular data.
