Chen Bo Calvin Zhang

I am the ML Research Ops Lead at Scale AI, where I focus on designing and running evaluations, developing benchmarks, and managing leaderboards.

Previously, I was a research intern at CHAI, where I worked with Micah Carroll on multi-turn red teaming strategies for large language models. I also spent time as a visiting scholar at MIT, where I focused on online learning and reward design for reinforcement learning, working with Zhang-Wei Hong, Aldo Pacchiano, and Pulkit Agrawal.

I received my MSc in Data Science from ETH Zurich, where I conducted reserach in preference-based reinforcement learning with Giorgia Ramponi.

Earlier, I completed my BSc (Hons) in Computer Science and Mathematics from the University of Manchester, with a focus on adversarial attacks in deep reinforcement learning under the supervision of Tingting Mu.

My research interests include evaluation methodologies, agents, sequential decision making, and AI safety and alignment.

Google Scholar / Twitter / GitHub / LinkedIn

News

Nov 20, 2025	I recorded a podcast! Check out the latest episode of Chain of Thought on PRBench
Nov 13, 2025	We published PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning, a new benchmark to measure how well LLMs do on prefessional domains
Nov 12, 2025	Check out our new paper ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Aug 28, 2025	We published Reliable Weak-to-Strong Monitoring of LLM Agents, extending our work on SHADE-Arena
Jun 19, 2025	Excited to share our new paper SHADE-Arena, in collaboration with Anthropic
Jan 27, 2025	I am excited to join Scale AI as a ML Research Ops Lead
Jan 22, 2025	ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization has been accepted at ICLR 2025

Latest Posts

Sep 16, 2024	The Quantum Cheese Conundrum

Selected Publications

ICLR

ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization

Chen Bo Calvin Zhang, Zhang-Wei Hong, Aldo Pacchiano, and Pulkit Agrawal

In The Thirteenth International Conference on Learning Representations, 2025

Bib HTML

@inproceedings{zhang2025orso,
  title = {ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization},
  author = {Zhang, Chen Bo Calvin and Hong, Zhang-Wei and Pacchiano, Aldo and Agrawal, Pulkit},
  booktitle = {The Thirteenth International Conference on Learning Representations},
  year = {2025},
}

arXiv

Humanity’s Last Exam

Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, and 2 more authors

arXiv preprint arXiv:2501.14249, 2025

Bib HTML

@article{phan2025humanity,
  title = {Humanity's Last Exam},
  author = {Phan, Long and Gatti, Alice and Han, Ziwen and Li, Nathaniel and Hu, Josephina and Zhang, Hugh and Zhang, Chen Bo Calvin and Shaaban, Mohamed and Ling, John and Shi, Sean and others},
  journal = {arXiv preprint arXiv:2501.14249},
  year = {2025},
}

arXiv

SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents

Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, and 3 more authors

arXiv preprint arXiv:2506.15740, 2025

Bib HTML

@article{kutasov2025shade,
  title = {SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents},
  author = {Kutasov, Jonathan and Sun, Yuqi and Colognese, Paul and van der Weij, Teun and Petrini, Linda and Zhang, Chen Bo Calvin and Hughes, John and Deng, Xiang and Sleight, Henry and Tracy, Tyler and Shlegeris, Buck and Benton, Joe},
  journal = {arXiv preprint arXiv:2506.15740},
  year = {2025},
}

arXiv

Reliable Weak-to-Strong Monitoring of LLM Agents

Neil Kale, Chen Bo Calvin Zhang, Kevin Zhu, Ankit Aich, Paula Rodriguez, Scale Red Team, Christina Q Knight, and Zifan Wang

arXiv preprint arXiv:2508.19461, 2025

Bib HTML

@article{kale2025reliable,
  title = {Reliable Weak-to-Strong Monitoring of LLM Agents},
  author = {Kale, Neil and Zhang, Chen Bo Calvin and Zhu, Kevin and Aich, Ankit and Rodriguez, Paula and Team, Scale Red and Knight, Christina Q and Wang, Zifan},
  journal = {arXiv preprint arXiv:2508.19461},
  year = {2025},
}

arXiv

MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More Than Outcomes

Yu Ying Chiu, Michael S Lee, Rachel Calcott, Brandon Handoko, Paul Font-Reaulx, Paula Rodriguez, Chen Bo Calvin Zhang, Ziwen Han, Udari Madhushani Sehwag, and 2 more authors

arXiv preprint arXiv:2510.16380, 2025

Bib HTML

@article{chiu2025morebench,
  title = {MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More Than Outcomes},
  author = {Chiu, Yu Ying and Lee, Michael S and Calcott, Rachel and Handoko, Brandon and de Font-Reaulx, Paul and Rodriguez, Paula and Zhang, Chen Bo Calvin and Han, Ziwen and Sehwag, Udari Madhushani and Maurya, Yash and others},
  journal = {arXiv preprint arXiv:2510.16380},
  year = {2025},
}

arXiv

ResearchRubrics: A Benchmark of Prompts and Rubrics for Evaluating Deep Research Agents

Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, and 2 more authors

arXiv preprint arXiv:2511.07685, 2025

Bib HTML

@article{sharma2025researchrubrics,
  title = {ResearchRubrics: A Benchmark of Prompts and Rubrics for Evaluating Deep Research Agents},
  author = {Sharma, Manasi and Zhang, Chen Bo Calvin and Bandi, Chaithanya and Wang, Clinton and Aich, Ankit and Nghiem, Huy and Rabbani, Tahseen and Htet, Ye and Jang, Brian and Basu, Sumana and others},
  journal = {arXiv preprint arXiv:2511.07685},
  year = {2025},
}

arXiv

PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, and 2 more authors

arXiv preprint arXiv:2511.11562, 2025

Bib HTML

@article{akyurek2025prbench,
  title = {PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning},
  author = {Aky{\"u}rek, Afra Feyza and Gosai, Advait and Zhang, Chen Bo Calvin and Gupta, Vipul and Jeong, Jaehwan and Gunjal, Anisha and Rabbani, Tahseen and Mazzone, Maria and Randolph, David and Meymand, Mohammad Mahmoudi and others},
  journal = {arXiv preprint arXiv:2511.11562},
  year = {2025},
}

ICML

HIP-RL: Hallucinated Inputs for Preference-based Reinforcement Learning in Continuous Domains

Chen Bo Calvin Zhang, and Giorgia Ramponi

In ICML 2023 Workshop: The Many Facets of Preference-Based Learning, 2023

Bib

@inproceedings{zhang2023hip,
  title = {HIP-RL: Hallucinated Inputs for Preference-based Reinforcement Learning in Continuous Domains},
  author = {Zhang, Chen Bo Calvin and Ramponi, Giorgia},
  booktitle = {ICML 2023 Workshop: The Many Facets of Preference-Based Learning},
  year = {2023},
}