It’s not easy being one of Silicon Valley’s favorite benchmarks.  SWE-Bench (pronounced “swee bench”) launched in November 2024 to evaluate an AI model’s coding skill, using more than 2,000 real-world programming problems pulled from the public GitHub repositories of 12 different Python-based projects.  In the months since then, it’s quickly become one of the most popular tests in AI. A SWE-Bench score has become a mainstay of major model releases from OpenAI, Anthropic, and Google—and outside of foundation models, the fine-tuners at AI firms are in constant competition to see who can rise above the pack. The top of the […]

There’s something that bothers me about the chatter that AI is making “intelligence” ubiquitous. For example, in a recent Bloomberg article, “AI Will Upend a Basic Assumption About How Companies Are Organized,” Azeem Azhar wrote: As intelligence becomes cheaper and faster, the basic assumption underpinning our institutions—that human insight is scarce and expensive—no longer holds. When you can effectively consult a dozen experts anytime you like, it changes how companies organize, how we innovate and how each of us approaches learning and decision-making. The question facing individuals and organizations alike is: What will you do when intelligence itself is suddenly […]

If your company is building any kind of AI product or tool, congratulations! You are now an AI company. Yes, you’re still a retail company. Or a bank. Or a CPG operation. You’re that plus an AI company—let’s call this an AI as Well company (AIAW)—granting you a license to tell sales prospects and investors that you’re “doing AI.” That license also puts you on the hook for new responsibilities. They’re easy to skip over at first, but you’ll hold yourself back from your true AI potential if you do. And maybe take on needless risk exposures in the process. […]