PhD Students in the AI Era: Blessing or Curse?

Opening

Dear reader, something pretty shocking happened in academia last week. ICML¹, the world’s largest machine learning conference, desk-rejected 497 papers in one sweep. The reason is unusual: it wasn’t a quality problem with the papers themselves — it was that the authors got caught secretly using AI while peer-reviewing other people’s papers.

They had explicitly agreed not to use AI — and used it anyway.

What makes this case interesting is how they got caught. The ICML organizing committee had embedded an invisible “watermark” into the review PDFs — invisible to human eyes. Hidden inside that watermark were instructions readable only by AI, and if an AI followed those instructions and inserted specific phrases into a review, that review got flagged. The phrases were drawn randomly, two per paper, from a dictionary of 170,000 phrases, making the odds of an accidental match less than 1 in 10 billion. This is actually a method I built myself a while back — being validated this way is gratifying, if also a little bittersweet…

But what struck me about this case was a more fundamental question than the detection technique itself. The fact that even researchers at the frontier of AI research depend on AI enough to break rules they agreed to — is that an individual ethics problem, or is something more structural at work?

Today I want to unpack this question across the whole research ecosystem, from PhD students to senior researchers.

Why ICML Researchers Broke the Rules

Let’s unpack the structure of the ICML case a bit more. ICML 2026 ran two policies on AI use. Policy A (conservative) banned AI use entirely during review, while Policy B (permissive) allowed AI for understanding papers and polishing reviews. Reviewers chose which policy they wanted to follow themselves. In other words, anyone who chose Policy A had personally promised not to use AI.

Here’s what happened. Among reviewers who chose Policy A, 506 were caught using AI, and 795 of their reviews (about 1% of all reviews) were struck. Of these, 51 reviewers had more than half of their submitted reviews written by AI, and they were stripped of their reviewer status entirely. Under ICML’s reciprocal review² policy, 497 papers (about 2% of all submissions) authored by rule-breaking reviewers were rejected.

One thing worth noting here: as the ICML committee itself acknowledged, this detection method only catches the most blatant violations — the kind where someone feeds the PDF of the paper they’re reviewing straight into AI and copy-pastes the output. The watermark’s existence was public knowledge for most of the review period, and anyone paying even a little attention could have avoided it. That 1% still got caught anyway suggests the real rate of AI use could be far higher. As I noted in an earlier blog post, that PDF watermark can be stripped out with a single pass of a Python script. And yet… people still submitted reviews like this to a conference as prestigious and international as ICML? It makes you wonder.

Indeed, in a 2025 survey by academic publisher Frontiers of roughly 1,600 researchers across 111 countries, 53% said they had used AI in peer review. Narrow that to early-career researchers (5 years or less of experience), and the figure jumps to 87%. But policy at many journals and conferences hasn’t caught up with this reality.

The PhD Student’s Dilemma

This same pattern is playing out at the student level too. In a Nature survey of roughly 3,800 PhD students, 75% said AI improves their efficiency, and 71% said using AI for academic work is acceptable. But at the same time, 81% said they don’t fully trust AI, and 65% worried it’s weakening their thinking, research, and writing abilities.

A survey by the Higher Education Policy Institute (HEPI) in the UK shows the pace of this shift even more starkly. Among UK undergraduates, AI tool usage jumped from 66% to 92% in a single year, and the share using AI in assessments rose from 53% to 88%. Josh Freeman, who led the survey, said behavioral change at this scale is “almost unprecedented.”

Stories from the field make this vivid. He Yinghui at Tsinghua University uses ChatGPT and Gemini every day but insists that “AI-generated code always has to be verified.” Richard Ang at the University of Western Australia once had an entire experiment fail after he let ChatGPT calculate fertilizer dosages — the AI had misunderstood the question itself. His takeaway: “AI will never tell you your design is weird. If you ask it to do something impossible, it just does it.”

Leona Diala, a PhD student modeling infectious disease at the University of Abuja, had a similar experience. She asked AI to describe a graph, and it said the values were increasing — but the graph actually showed a decrease. “You need to already understand what you’re asking about before you use AI, or you won’t catch the mistake.”

The core point, in the end, is this: AI executes fast, but it doesn’t exercise judgment. And judgment is precisely the core capability a PhD is supposed to build.

“Cognitive Debt” — The Brain Is Already Responding

There’s a reason this issue isn’t just an ethical debate over “how much AI use is acceptable.”

The results of a 2025 preprint³ by Nataliya Kosmyna’s team at MIT Media Lab, titled “Your Brain on ChatGPT,” are quite striking. They split 54 participants into three groups — one wrote essays with no tools, one used a search engine, and one used ChatGPT — and measured brain activity with EEG⁴.

The results were unambiguous: the more participants relied on external tools, the more systematically their brain’s functional connectivity weakened. The no-tool group showed broadly distributed activation across regions tied to creativity, language, planning, and working memory, while the ChatGPT group showed the weakest connectivity patterns of the three.

What’s more interesting is the follow-up experiment. Participants who had used ChatGPT and then switched to writing without tools showed reduced alpha- and beta-wave connectivity. The research team called this “cognitive debt” — the idea that delegating cognitive work to AI feels convenient in the moment, but the cost accumulates as weakened neural connectivity. Many participants in the ChatGPT group couldn’t even remember the essay they had just “written.” Their sense of ownership over the writing was also the lowest of the three groups.

Of course, this study hasn’t yet been peer-reviewed, and a sample of 54 is small. Kosmyna herself cautioned against concluding flatly that “AI is harmful.” But the direction is clear: outsourcing cognitive work to AI can weaken the brain’s “muscle” for that work.

Connect this to the ICML case, and a structure emerges. Researchers who promised not to use AI and then used it anyway may not simply have been lazy. Once a pattern of delegating cognition to AI takes hold, going back becomes cognitively costly in itself.

So What Needs to Be Protected

University responses still haven’t caught up with this shift. In a European University Association (EUA) survey of 217 European universities, only 5% said their existing AI guidelines were adequate. 38% were drafting a policy for the first time, and 13% had no policy at all. Korea is no different. In a survey by the Korea Education and Research Information Service (KERIS), more than half of instructors said “student assignments are becoming excessively formulaic,” yet institutionalized AI-use standards written into university regulations remain very rare.

So in this landscape, what should a PhD actually be cultivating?

Amina Yonis of the academic support firm The Page Doctor offers a practical distinction: use AI for literature search and organization, but do your own data analysis. The same goes for writing. “If you let AI write the first draft, it’s hard to break out of that frame. It’s better to write it yourself first, then use AI to polish it.”

On a broader level, computational neuroscientist Natalia Bielczyk’s view is compelling. Don’t compete with AI on speed or memory — focus instead on what machines still can’t do: designing good questions, navigating ambiguity, and devising ways to test ideas against the real world. The true superpower of a PhD, she argues, is internalizing “a systematic approach to problem-solving,” and AI should be a tool that frees up more time for slow, conceptual work.

On the other end, some researchers reject AI altogether. Samuele Ramellini, a PhD student in avian behavior at the Australian National University, first used AI to make a graph last November — two weeks later, he couldn’t remember the code or even what he had asked for. By contrast, he still remembers every graph from his undergraduate thesis, ten years on. “AI gives you instant reward, but you can’t learn from it.” He even refuses to feed research ideas into AI. “Paper, sticky notes, a whiteboard — those are so much better.”

Alex Sen Gupta, a climate scientist at UNSW, posed a sharper question in a 2025 Nature essay. He argues for a full redesign of PhD training, while acknowledging the uncertainty involved: “No one can predict how this field will develop. A year from now, two years from now, things could look completely different.” He notes that tech companies will keep closing whatever cognitive gaps AI currently has, as soon as those gaps are found.

Professor Manikandan Palanichamy of Østfold University College in Norway zeroes in on the timing of the issue. “Students need ethical guidance on responsible AI use from early in their education. Otherwise, we risk producing a generation of researchers who know how to operate AI tools but lack the foundational capabilities to make genuine breakthroughs.”

Oz’s Lens

Watching the ICML case unfold, two things came to mind.

The first is a pattern I’ve seen countless times while building go-to-market strategies: the “tool adoption curve.” When a new tool is introduced, the early framing is almost always “efficiency gains.” That was true of CRM, and true of marketing automation. But the real issue isn’t the tool itself — it’s which behaviors the tool makes easy, and which it makes hard. AI has made it extremely easy to “skim someone else’s paper and produce a plausible-sounding review.” And it has made it relatively harder — more precisely, more costly — to “read a paper slowly and build your own critical perspective.”

The second is that ICML is being held in Seoul this July. This case isn’t happening in some distant country — it took place at a conference where Korean AI researchers participate as both reviewers and authors. And the reality in Korean graduate schools isn’t much different. A 2025 survey published in the Journal of the Korea Academia-Industrial Cooperation Society found that 85.7% of Korean university students were aware of ChatGPT, and 89.3% had actually used it.

What I find most structurally significant in all this is the thread connecting the MIT study’s “cognitive debt” to ICML’s “rule violations.” The researchers who broke their promise may not simply have been lazy. Once you’ve adapted to cognitive convenience, going back itself becomes costly — the same way finding your way with a paper map becomes painful after you’ve gotten used to GPS navigation.

That means the advice to “focus on what AI can’t do” needs reconsidering too. As Sen Gupta notes, the territory AI can’t cover will keep shrinking. A more durable strategy, in my view, is this: instead of asking what AI can’t do, define which cognitive muscles need to be maintained regardless of whether you use AI or not. Critical verification, designing original questions, judgment under uncertainty — these don’t lose value no matter how the tools change.

The problem is that AI is replacing the very training process that builds these muscles. That’s not a tool problem — it’s a problem of educational design. And as ICML showed us, this isn’t just a student problem — it’s a problem for the entire research ecosystem.

Closing

• AI boosts research efficiency, but it’s eroding the core capabilities that make a researcher a researcher — independent thinking, critical judgment, and original question design.
• This isn’t just a student phenomenon. The ICML case showed that even senior researchers aren’t free from the cognitive inertia of AI dependence.
• University and conference systems haven’t caught up with this shift, leaving individuals to find their own balance.

Yonis’s advice turns out to be surprisingly practical.

“AI has only been around for about three years. Most people finished their PhDs without it. You can too.”

Maybe the key question isn’t “whether to use AI or not” — maybe it’s Ramellini’s question.

“You saved an hour, thanks to AI. So what are you going to do with it?”

---

📎 References & Further Reading

• Nature, “AI and the PhD student: friend or foe?”, Nature 651, 842-844 (2026). : The main starting point for today’s issue. Covers how PhD students worldwide are actually using AI, through interviews.
• ICML 2026 Program Chairs, “On Violations of LLM Review Policies”, ICML Blog (2026.3.18). : The official statement on the 497-paper rejection incident. Includes a technical explanation of the detection method and false-positive rate (0.0001) data.
• Nature, “Major conference catches illicit AI use — and rejects hundreds of papers”, Nature (2026.3.25). : Nature’s follow-up report on the ICML case, with academic reactions and context well summarized.
• Kosmyna, N. et al., “Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task”, arXiv:2506.08872 (2025). : The MIT Media Lab’s EEG study showing systematic weakening of brain connectivity with AI use. Note this is a preprint (not yet peer-reviewed).
• Sen Gupta, A., “PhD training needs a reboot in an AI world”, Nature 647, 27-28 (2025). : An essay arguing for a redesign of PhD training. The appendix — an experimental exploration of the future of PhDs in the AI era through a simulated dialogue with Claude (Opus 4.1) — is a fascinating read.
• Frontiers, “Unlocking AI’s untapped potential: responsible innovation in research and publishing” (2025). : A survey of 1,645 researchers across 111 countries — the original source for the 53% AI-use-in-peer-review figure.
• European University Association, “Policies in doctoral education” (2026.1). : The state of PhD AI policy across 217 European universities. The finding that only 5% considered their policies adequate stands out.
• Freeman, J., “AI in assessments: UK undergraduate survey”, Higher Education Policy Institute (2025). : The original source for the surge in UK undergraduate AI usage figures.
• Kwangseob Ahn, Vibe Coding Misadventures EP.01, PDF AI SHIELD, 2025.05.23. : A product I built last year. I ended up selling the whole solution to someone. :)

The author, Kwangseob Ahn, is a professor of business administration at Sejong University and lead consultant at OBF (Oswarld Boutique Consulting Firm). He teaches statistics and data analysis — business data management and business analytics — while leading GTM and AI strategy consulting in the field, designing the seam between technology and business. He has published academic research on a memory architecture for AI dialogue systems (HEMA) and runs Daily Arxiv, a daily curation of global AI papers. He holds a master’s from Korea University’s Graduate School of Technology Management and a KMBA. He is the author of Homo Brainless: The People Who Outsource Their Thinking.

Footnotes

1. ICML (International Conference on Machine Learning): one of the world’s largest machine learning conferences, considered one of the “big three” in AI alongside NeurIPS and ICLR. In 2026 it will be held in Seoul in July. Total submissions exceed 20,000 papers, which puts a correspondingly heavy strain on the peer review system. ↩

2. Reciprocal Review: a system requiring authors who submit papers to a conference to also review other people’s papers. It was introduced to address a shortage of reviewers. At ICML, this creates a joint-liability structure: if a reviewer breaks the rules, their own paper gets rejected too. This structure is why 497 papers were rejected in this incident. ↩

3. Preprint: a version of a research paper posted to an open server like arXiv before it undergoes peer review. It allows the latest research to be shared quickly, but readers should keep in mind that it hasn’t been fully vetted. ↩

4. EEG (Electroencephalography): a method that measures the brain’s electrical activity in real time via electrodes attached to the scalp. It has lower spatial resolution than fMRI but higher temporal resolution, making it well-suited to tracking how the brain responds during specific cognitive tasks. ↩