Rishub Jain and Josh Jacob

We’re starting a nonprofit to build Scalable+Human Oversight! We left Google DeepMind to start this org, having worked on key parts of this research agenda over the past years. This blog lays out our research plan. And, we’re hiring!! 🙂

First 6 months of our org

Initial Goal: Build better judges

Definition of a Judge: Human or AI systems that verify correctness and alignment of an AI in an interaction

Where improved judges can help:

  1. Imperfect judges currently used in training lead to reward hacking — i.e. AI learning the wrong goals. Examples:
    1. Human decisions used in RLHF reward sycophancy (prioritizing agreeableness over factual accuracy), and LLMs reflect these priorities [Sharma et al. (2023)]
    2. Coding RL environments can be completed by cheating, and LLMs exhibit some deceptive cheating behavior on software tasks [Baker et al. (2025)]
  2. Evaluation done with imperfect judges lead to misleading research results [Gema et al., 2025; Panickssery et al, 2024]
  3. Deployment-time Monitoring for alignment is bottlenecked on judge quality [Kutasov et al. (2025)]
  4. Accurate judges can improve the quality of all other data used in the AI development and deployment stack: SFT, RL-Environments, Red-teaming, Pre-training, etc.

How:

  1. Build and maintain a diverse and comprehensive leaderboard for judges.
    1. Composed of >20 subdatasets, across areas like deception detection, cultural bias, and unsafe agent action.
    2. In addition to static evaluations, we will also include evaluations where the judges are subject to direct optimization pressure from the models they are overseeing during RL
  2. Hill-climb the leaderboard
    1. Focus on methods that leverage the complementary strengths of Humans and AIs (i.e. Human-AI Complementarity). E.g.:
      1. Improve AI’s confidence estimation, and defer to humans when less confident
      2. Train a router to best delegate tasks to either humans or AI
      3. Break down a task into sub-tasks, and route subtasks (based on e.g. above)
      4. Provide task-specific assistance UIs for humans (e.g. generated on the fly)
    2. Methods should generalize to all diverse datasets in our leaderboard.
    3. Will benchmark and compare with all other judge methods (e.g. debate, LLM-as-a-judge).

Why focus on Human-AI Complementarity

Although many others working in Scalable Oversight (i.e. the field of improving data and evaluation quality of AI, as AI rapidly becomes more capable) recognize the importance of involving Humans in the process, they often study their methods with LLMs as substitute to humans — to both make the research iteration easier and cheaper, and also because many see humans playing a smaller role in the development AI as AI-alone can do many tasks as well or better. We share similar views too, but think that Humans will have complementary strengths to AIs for many years to come:

  • Jaggedness: Even after we get superintelligent AIs that are more accurate than humans on average, we believe AI’s intelligence is and will continue to be jagged, and we can find places where humans can still add value.
  • Robustness: Humans and AIs might cover each other’s weaknesses against optimization pressure faced by judges in training, even if the human-AI team is not better overall (e.g. the comprehensiveness-hallucination trade-off described in McAleese et al., 2024).
  • Collusion: Future AI-only judges used in supervision might deceptively collude with the base model being trained. As eval awareness becomes more prominent, it’s already hard to know if this collusion is happening. Oversight from trusted humans adds a layer of defense.
  • Unseen Human Values: AI systems might become good at verifying aligned behavior in out-of-distribution scenarios, but humans might be needed in completely new situations. And as humans’ values evolve, especially in our rapidly changing society, updated values will need to be understood from real humans.

Once we have better Judges

We will deploy our better judges in frontier AI lab training, evaluations, and monitoring. Answering questions like:

  • Does our improved judge actually reduce reward hacking in RL?
  • How to best incorporate the slower and more expensive humans, finding the optimal trade off between cost, speed, and quality?
  • When and how to re-train our judge methods during RL, as the base model policy evolves to hack the judge?
  • Initially our “Humans” will be paid annotators. How does our research transfer to:
    • The human user of an agent, in the real-time monitoring of that agent?
    • AI researchers, who verify the quality of data and processes overall, and assess validity of alignment methods?
  • And more…

And, expand beyond “classifier” judges towards improving the output of generative tasks:

  • Most importantly, developing the specifications for judges (including our human/AI protocol) to improve the results of RLAIF and strengthen evaluation quality:
    • High-level general and system instructions given to judges
    • Fine-grained per-example rubrics used by judges to evaluate individual tasks
    • Synthetic documents generated from Constitutional AI specifications
  • We’ll include these datasets to datasets for these generative tasks in our leaderboard, and use a similar hill-climbing approach to improving these methods.

FAQ

  • Funding: We have received a $11M grant from Coefficient Giving ($4M of which is pending due diligence).
  • Name: Sampura Research comes from the Sanskrit word सम्पूरक (sampūraka), which means “complementary” or “one who fills up, completes, or makes full”, reflecting our focus on Human-AI Complementarity.
  • Prior work: We have been pioneering this research agenda since 2023 — first at DeepMind (paper), then across successive SPAR and MARS fellowships with cohorts of 10-20 mentees (first paper, second to come next month). Our nonprofit is a direct continuation, scaled up dramatically.
  • Dual use: The field of Scalable Oversight, and the pursuit of improving judges and data quality, is fundamentally dual-use: it may both accelerate capabilities of AI, and make it safer. We aim to differentially help safety by making the majority of this leaderboard alignment-focused, where improved judging more directly translates to improved alignment. By maintaining such a leaderboard and being actively involved in its datasets’ development, we also hope to bring together and strengthen the alignment dataset community. At the same time, we still need to ensure our methods transfer to capabilities-focused datasets, and will incorporate some in our leaderboard. This is because reward-hacking in capabilities tasks like general coding might lead to egregiously misaligned models (MacDiarmid et al., 2025).
  • Leaderboard makeup: AI Alignment is an incredibly diverse objective. Existential-risk-focused benchmarks often center around detecting deceptive or scheming behavior from rogue AI. But a verifier asking “is this AI causing harm?” also has to catch issues like cultural bias, unsafe interactions with children, and companion-AI dependence. Research that focuses on single-domain may lead to methods that don’t generalize, like our previous “Evidence-only” assistance method that only applied to the retrieval tasks we were studying. We aim to keep our leaderboard diverse and comprehensive, and continuously updated as datasets get saturated.
  • Why us: Rishub co-led this research agenda at DeepMind. Since 2023, he has been working directly on this line of work, including trying to deploy the research internally. Josh co-led work on human data, understanding how to make the most effective use of data vendors and tooling that LLM companies use, and how both AI and Human data is used in training and evals.
  • Debate?: AI Safety via Debate is one of the most explored Scalable Oversight methods in the AI Safety community. Research on debate has focused primarily on understanding the right protocol to train the “debate assistance” for the judge. We are excited to test if these findings hold on our leaderboard when using Human judges, and how to best combine our advances in Human-AI Complementarity into Debate.
  • Acknowledgements: Thank you to Konstantinos Voudouris, David Africa, and Alex Adams for helpful feedback on this blog.