Analyst, Applied AI at HeyMilo

HeyMilo is building AI interviewers that automate and improve hiring through conversational AI. We work closely with companies to bring AI into real hiring workflows. We also run an applied AI team that studies where today's models succeed and fail across verticals.

The Role

We're hiring an Analyst to join our Applied AI team in Colombo. You'll design the tasks, rubrics, and datasets we use to evaluate AI systems on real-world workflows, and define the ground truth that decides whether a model's attempt counts as correct.

The quality of everything we build downstream depends on the quality of the tasks and ground truth you produce.

What you'll do

• Design realistic, multi-step tasks that test AI systems on real-world workflows, and author the grading rubrics that score them

• Build and curate ground-truth datasets from real operational data and domain research

• Review model attempts against your tasks: analyze failure modes, catch scoring loopholes, and identify where models pass for the wrong reasons

• Calibrate task difficulty so results separate models in a meaningful way

• Analyze score distributions across models and turn results into clear, defensible findings

• Contribute to research write-ups and published evaluations alongside our engineers and academic reviewers

What we're looking for

• Master's or PhD in AI, Machine Learning, Computer Science, Statistics, or a closely related quantitative field

• Strong understanding of how large language models are trained and evaluated

• Hands-on experience assessing LLM outputs: evals, annotation, red-teaming, or benchmark work

• Comfortable with Python and data tooling for building datasets and analyzing results

• Exceptional attention to detail and intellectual honesty; you flag a broken task even when the scores look good

• Clear written communication; you can explain a subtle failure mode to both engineers and non-technical readers

Bonus

• Familiarity with reinforcement learning concepts (reward design, verifiable rewards)

• Experience contributing to published research or open evaluation suites

• Domain knowledge in recruiting, staffing, or other operations-heavy industries

Why join

• Ground-floor role on a new applied AI team with direct ownership of what we measure and publish

• Work reviewed by and published alongside experienced researchers

• High visibility, fast-paced, execution-driven environment

• Competitive pay and benefits