Zurich Tested 100 Students on Vibe Coding, No Code Shown

100 People Sat in Front of Screens Showing Zero Lines of Code

100 students sat down in front of screens showing zero lines of code, in a lab in Zurich. On the left was a chat window, on the right a live preview of the app being built, and a 15-minute timer sat in between.

There were three tasks. First, try out an app that was already running, then rebuild it from scratch. Second, add one more feature to it. Third, clone yet another app whose labels had been stripped so its purpose was impossible to guess. All a participant could do was type prompts. The screen where the model would normally spit out code was deliberately blurred—just clear enough to show that something was happening—so nobody could actually read it.

Participants were paid 55 Swiss francs for a session that ran 1 hour and 45 minutes. The only people selected had completed an intro computer science course and had some prior experience programming with LLMs—not complete beginners, but not working developers either.

The reason researchers at ETH Zurich pre-registered¹ and ran this experiment is simple: building software without touching a single line of code—what people commonly call vibe coding—is spreading fast, and yet nobody had ever actually measured who’s good at it.

The results came out in March, and a summary of them has recently been making the rounds on LinkedIn. That summary turned a sentence that doesn’t exist in the paper into its headline conclusion. And one number that genuinely does appear in the paper was left out entirely.

---

The People LinkedIn Called “100 Developers”

The gist of the summary circulating online goes like this: ETH tested 100 developers in a production-grade environment, the results were brutal, computer science knowledge was an overwhelming predictor of performance, and ultimately, vibe coding isn’t about writing code—it’s about debugging logic.

Lined up against the actual paper, the two diverge sharply.

Summary	Paper
100 developers	100 students from ETH Zurich and the University of Zurich. Average age 25; 51 from engineering/technology, 20 from natural sciences, 11 from humanities and social sciences
Production-grade environment	An experimental platform the researchers built themselves, modeled on commercial tools—but different from them in that it showed zero code
Overwhelming predictor	The correlation between CS achievement2 and performance was 0.39. A model adding writing ability explained 20.8% of the variance
Brutal results	Average vibe coding score was 0.45 out of 1. Everyone struggled
Debugging logic by reading code	Participants couldn’t see any code at all. The authors explicitly reject causal claims and present problem-decomposition ability only as a hypothesis

One part checks out. CS achievement really did carry roughly twice the unique explanatory power that writing ability did. Adding CS achievement after writing ability raises explanatory power by 12.5 percentage points; reversing the order raises it by 5.9 points.

But the relationship between writing ability and performance shifted once other factors were accounted for. Once general cognitive ability³ was controlled for, the link between writing ability and performance lost statistical significance—a partial correlation⁴ of 0.186, p-value 0.066. Under the same conditions, the relationship between CS achievement and performance remained statistically significant.

And there’s one number that shows up not in the paper’s main results but buried in its exploratory analysis.

Participants’ self-reported frequency of LLM use was negatively correlated with vibe coding performance—a correlation coefficient of −0.258, p-value 0.010. What does that mean? It means the people who said they used LLMs more often actually built worse apps. The same pattern showed up with writing scores, at a negative correlation of −0.282. CS achievement, by contrast, showed essentially no correlation at all—0.001.

Which Metric Actually Correlated with Assignment Performance

Let me first sort out how to read that −0.258. The authors themselves wrote that they were surprised by this result, and they laid out three explanations side by side: LLMs may have degraded writing ability, people who were already weak writers may have relied more heavily on LLMs, or both. This is a self-reported measure, the sample is 100 students, and it’s a post-hoc exploratory finding rather than a pre-registered confirmatory hypothesis. So we shouldn’t turn this number into the sentence “heavy AI use makes you dumber.” That would just repeat, in the opposite direction, the same mistake the summary article made.

There’s evidence the research team designed this carefully. They alternated the order — half the participants took the survey first, half did the assignment first — and the two groups’ average performance came out to 0.446 and 0.448, essentially identical. That means writing the essay first didn’t somehow make people better at writing prompts. One pre-registered hypothesis was flatly rejected, too: they expected that the more a task’s context was stripped away, the more writing ability would matter, but in practice there was no difference from the feature-addition task. For reference, among the three tasks, the decontextualized one actually scored lowest — 0.34 out of 1.

One more caveat needs attaching. The model used in the experiment was the May 2025 version of Claude Sonnet 4. When a model shifts generations, the degree to which it autonomously fills gaps in a prompt changes, and that in turn shakes up the relationship between writing and performance. The numbers in this study are the result of an experiment run on a model frozen at a particular point in time.

Looking at this result, frequency of use alone is a poor proxy for AI competency. At least in this student sample, people who used AI more often didn’t produce better output.

The metric that showed the strongest correlation with assignment performance in this study was the quality of the prompts participants actually wrote.

The quality of the prompts participants actually wrote correlated with performance at 0.479. That’s the largest coefficient in the entire paper. The language-center instructors who built the essay grading rubric separately designed a rubric for prompts, and a single expert with no access to any other data scored them blind, evaluating consistency, task-appropriate complexity, and clarity of instruction.

The mediation analysis⁵ results are even clearer. Prompt quality mediated about 52% of the path from writing ability to performance, and once prompt quality was entered into the model, the direct effect of writing lost its significance. In this analysis, prompt quality explained a substantial share of the relationship between writing and performance. That said, a mediation analysis from a single cross-sectional study doesn’t prove causation or establish the sole operating pathway. It’s neither “good at coding = good at using AI” nor “good at using AI = good at writing.”

They also cross-checked this mechanically. They computed two automated metrics of lexical diversity in the prompts, and people who wrote prompts with more varied vocabulary scored higher both on the essay and on the assignment. In other words, the human-graded results and the machine-counted results pointed in the same direction.

One more thing. CS achievement and writing ability were not correlated with each other. 0.126, p-value 0.213. In this sample, no statistically significant correlation between the two scores was found. That doesn’t mean one person can’t possess or develop both abilities together.

So Why Are Organizations Counting Usage, of All Things?

This is where the original paper ends and my interpretation begins.

Right now, most organizations rely on three numbers when reporting AI adoption. Account counts, weekly active users, token consumption. All three have one thing in common: they’re just variations on usage frequency. They’re easy to count, they come straight out of the console, and when they go up, adoption looks like it’s going well.

What makes this experiment uncomfortable is that in this sample, the metric came out with the opposite sign. Of course, you can’t directly transplant a 15-minute lab task with 100 students onto real corporate settings. But if we’re setting budgets and staffing based on a metric that points the wrong direction, we should at least check, within our own organization, what the actual relationship between usage and output quality looks like.

I think one reason usage metrics keep getting used is that they’re easy to manage. Account count changes the moment you sign a bigger contract, but output quality has to be evaluated separately. That’s why we need to look at the numbers that report adoption status alongside the numbers that actually measure performance.

We can measure usage and output quality separately. The organizational application ideas below should be distinguished from the paper’s direct findings.

What We Count Now	What This Study Suggests
Account count, active users	An organizational adoption metric. What the study directly measured was individuals’ self-reported LLM usage frequency, which correlated with performance at −0.258
Token consumption	Not directly tested in this study. An item to compare separately against organizational output quality
Time spent using the tool	Should be distinguished from this study’s self-reported usage frequency
Prompt quality	Correlated with performance at 0.479, the highest coefficient in the study
Grading the output	What the authors actually used — a 4-point rubric

Measuring prompt quality might sound unrealistic, but the paper published its rubric as supplementary material. It has three axes: consistency, appropriateness relative to task complexity, and clarity of instructions — a level of detail that could be bolted onto the code reviews or proposal reviews teams already run.

For Korean organizations, this contrast stings a bit more. As I discussed in Issue 206, Korea already spends more hours glued to AI than other countries do. We’ve been assuming that time equals competence — and this study, at minimum, puts a question mark next to that assumption. The point is to look not just at hours logged, but at the actual quality of the prompts written and the outputs produced.

What you can try right now doesn’t have to be big. Pick ten AI-generated outputs your team produced last quarter, pull up the prompts that were used for them, and score them on the three axes. That gives you a starting point for comparing how prompt scores, output scores, and usage relate to each other. Rather than treating ten cases as proof of an organization-wide pattern, it’s better to align your scoring criteria and grow the sample from there.

Oswarld’s Lens

I think the real discovery in this paper isn’t the correlation coefficient — it’s the experimental design.

The researchers treated prompts not as outputs, but as data. They took the full set of prompts left behind by 100 people, treated it as a single unit, built a rubric, and applied blind scoring. Most organizations just let prompts pass through unexamined. They sit in logs, but nobody reads them, nobody scores them. The reason we can’t measure AI competency isn’t that we lack a measuring tool — it’s that we never treated the thing we should be measuring as data in the first place.

Once you start treating prompts as data, something else follows. Well-crafted prompts start getting reused across the team. A prompt that would otherwise stay buried in someone’s individual chat window suddenly has a reason to be reused by the team once it’s evaluated and shared. Which means, in effect, that prompts only get better through a repeated cycle of sharing and refining — and it leads to the conclusion that the so-called universal prompt that works everywhere doesn’t actually exist.

The second thing that caught my eye was the passage noting no significant correlation between CS achievement and writing scores. Reading that, I thought it might be worth evaluating the two abilities separately. On a team, someone who’s good at breaking down structure might collaborate with someone who’s good at putting intent into clear sentences. That said, the effect of team composition on performance isn’t something this study directly tested.

And finally, what actually pushed me to write this piece wasn’t the −0.258 — it was the fact that number disappeared from the summary. What survived was “so learn CS.” Only the number that fit the message someone already wanted to deliver made it through. By the time one paper turned into a single LinkedIn card, an explanatory power of 20.8% had become a “brutal result,” and an experiment where the code was never even visible had become “logic debugging.” And honestly, this is a message that plays well with existing CS majors and professionals already in the field. On LinkedIn, it’s content that sells.

Closing

Where you land in this depends on where Reader sits right now.

If you’re the one reporting on AI adoption, a good starting point is swapping one usage metric for one output-grading metric in your next report. Account counts justify budgets, but they don’t justify claims about capability.

If you’re hiring or assigning people, you can evaluate CS knowledge and writing ability separately. Just don’t generalize this sample’s finding — that the two scores weren’t significantly correlated — to everyone.

If you’re building this yourself, the most practical advice is to reread your own prompts. In this study, what best predicted performance wasn’t how long someone had used the tool, but what they wrote and how.

---

💬 What number are you currently using to report AI adoption results on your team? I’m curious whether you feel that number actually captures the quality of the real output.

📨 If you have a colleague who’s setting this quarter’s AI adoption metrics, pass this along. It’s worth a read before anyone starts counting accounts.

References & Further Reading

Primary sources

• Sverrir Thorgeirsson, Theo B. Weidmann, Zhendong Su, “Computer Science Achievement and Writing Skills Predict Vibe Coding Proficiency”, arXiv, March 14, 2026. Accepted at CHI 2026. Every figure in this issue comes from this paper. Section 9’s exploratory analysis in particular is rarely cited in summaries, so I’d recommend reading it directly
• Preregistration document, AsPredicted. If you want to distinguish which hypotheses were registered in advance from which results emerged as post-hoc exploration, check this document first

Background

• The CS achievement measure the paper uses is a 12-item shortened version of the SCS1, developed by Parker et al. It’s a test written in pseudocode, without assuming any particular programming language
• The term “vibe coding” itself was coined by Andrej Karpathy in February 2025. It matters for interpreting the results that this paper adopts his original definition — forgetting the code even exists — without modification

Related issues

• Issue 206, “Korea’s AI Usage Time Is Double America’s”
• Issue 223, “Is 92% Productivity a Real Result or a Performance”
• Issue 99, “What It Means That Everyone Can Now Code”

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. Preregistration: The procedure of registering hypotheses, sample size, and analysis methods in a public repository before an experiment begins. It’s a safeguard against changing hypotheses after seeing the results, so you need to read registered confirmatory analyses separately from post-hoc exploratory ones. ↩

2. CS achievement: In this study, this refers to the score on a 12-item shortened version of a test called the SCS1. It covers three categories — definitional knowledge, code tracing, and code completion — four items each. ↩

3. General cognitive ability: Measured using a public test called ICAR16. It consists of 16 items across four categories: verbal reasoning, matrix reasoning, letter series, and 3D rotation. ↩

4. Partial correlation: The correlation remaining between two variables after removing the influence of a third variable. Here, it’s the value after removing the effect of general cognitive ability. ↩

5. Mediation analysis: A method for estimating how much a variable sitting between A and B serves as a pathway when A influences B. Here, prompt quality sits between writing skill and performance. ↩