Explore AI roles

Keyboard friendly • Copy, share & save roles

Sort by

We are seeking an experienced Senior AI Evaluation Scientist to design and lead rigorous evaluation programs for pred...

$140k - $190k · Senior Fully remote

Apply

As a Cloud Architect, you will work within our growing DevSecOps practice delivering features to support cloud migrat...

Machine Learning Ai

$130k - $190k · Senior Fully remote

Apply

Steampunk is seeking an experienced Azure Search AI Interface Developer to design, develop, and maintain AI-powered s...

Remote Ai Ai Engineer Nlp

$120k - $190k · Senior Fully remote

Apply

In today’s rapidly evolving technology landscape, an organization’s data has never been a more important aspect in ac...

Machine Learning Data Science Data Engineer Ai

$125k - $160k · Senior Fully remote

Apply

Selected role

Senior AI Evaluation Scientist

Steampunk McLean, VA, US

Live

$140k - $190k · Senior Full-time Posted 14 days ago 4 min read

Overview:

We are seeking an experienced Senior AI Evaluation Scientist to design and lead rigorous evaluation programs for predictive and generative AI systems across our enterprise and client engagements. This role is critical to ensuring that AI solutions are accurate, reliable, safe, and aligned with mission outcomes. The Senior AI Evaluation Scientist will develop evaluation frameworks, build automated testing pipelines, and act as a subject-matter expert on AI quality, risk, and performance measurement. This role blends deep technical expertise with analytical rigor, experimentation, and cross-functional collaboration.

Contributions:

Lead the design and implementation of comprehensive evaluation frameworks for generative and predictive AI models, including accuracy, robustness, relevance, trustworthiness, fairness, hallucination rates, and safety.
Develop and maintain automated evaluation pipelines that continuously audit model outputs, monitor quality drift, and validate alignment with mission-specific constraints.
Create custom benchmark datasets, challenge sets, and adversarial evaluation strategies tailored to client domains and regulatory requirements.
Conduct in-depth error analysis, model behavior studies, and sensitivity assessments to inform iterative improvements in prompts, retrieval systems, models, and orchestration frameworks.
Partner with AI Product Engineers, LLMOps Engineers, and Data Scientists to drive model improvements through structured experimentation, A/B testing, and scientifically grounded evaluation cycles.
Advise teams on measurement methodologies, statistical significance, and best practices for Trustworthy AI evaluation in alignment with NIST AI RMF, MLSecOps, and agency governance requirements.
Document evaluation results, risks, and findings for technical and non-technical audiences, including engineering teams, leadership, and government clients.
Contribute to the development of standardized tools, reusable templates, and evaluation components to improve repeatability and quality across engagements.
Stay informed of advances in LLM assessment, safety science, red-teaming methodologies, and evaluation frameworks emerging from academia and industry.
Mentor junior evaluation staff and help grow Steampunk’s AI measurement and evaluation capabilities.
You will contribute to the growth of our AI & Data Exploitation Practice!

Qualifications:

Ability to hold a position of public trust with the U.S. government.
Bachelor’s, Master’s, or Ph.D. in Computer Science, Statistics, Machine Learning, Cognitive Science, Human-Computer Interaction, or a related field.
8+ years of experience evaluating machine learning, NLP, or generative AI systems, with strong familiarity with LLMs and retrieval-based architectures.
Deep understanding of evaluation metrics, statistical testing, dataset construction, experimental design, and model validation methodologies.
Hands-on experience with Python and libraries such as PyTorch, Hugging Face, LangChain, scikit-learn, and evaluation tooling (LLM-as-a-judge, rubric-based evaluators, or custom harnesses).
Demonstrated experience designing automated evaluation pipelines and integrating them into CI/CD or LLMOps workflows.
Strong understanding of AI governance, responsible AI principles, bias detection, fairness metrics, and risk identification.
Experience working with structured and unstructured datasets across multiple modalities (text, tabular, documents).
Familiarity with vector databases, RAG architectures, and multi-step LLM workflows.
Excellent analytical, written, and verbal communication skills, with the ability to translate evaluation insights into clear technical recommendations.
Proven ability to collaborate with cross-functional engineering and product teams while independently driving evaluation strategy.
Experience working in agile or iterative development environments and documenting scientific processes clearly.

About steampunk:

Steampunk relies on several factors to determine salary, including but not limited to geographic location, contractual requirements, education, knowledge, skills, competencies, and experience. The projected compensation range for this position is $140,000 to $190,000. The estimate displayed represents a typical annual salary range for this position. Annual salary is just one aspect of Steampunk’s total compensation package for employees. Learn more about additional Steampunk benefits here.

Identity Statement

As part of the application process, you are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud.

Steampunk is a Change Agent in the Federal contracting industry, bringing new thinking to clients in the Homeland, Federal Civilian, Health and DoD sectors. Through our Human-Centered delivery methodology, we are fundamentally changing the expectations our Federal clients have for true shared accountability in solving their toughest mission challenges. As an employee owned company, we focus on investing in our employees to enable them to do the greatest work of their careers – and rewarding them for outstanding contributions to our growth. If you want to learn more about our story, visit http://www.steampunk.com.

Ready to apply? You'll go straight to the hiring team.

No agency in the middle · Your profile is shared privately.

Explore the complete AI ecosystem

Discover the best AI-first companies, podcasts, events, learning paths, and jobs in one focused hub.

AI Learning Hub · Always curated, never noisy

Simple pricing for serious AI hiring

Pay per listing. No subscriptions, no hidden fees. Designed for teams hiring for high-impact AI roles.

No-risk: replace or refund if the role doesn’t get traction.

Standard listing

Reach active AI talent

For startups

$349 per role

30-day live listing in the AI job feed.
Structured posting written for AI engineers (we’ll help refine copy).
Instant alerts to subscribed candidates for your tags (LLMs, agents, infra, etc.).

Most teams see first applicants within 24–48 hours.

Max visibility to senior talent

For scale-ups

$649 per role

Everything in Standard, plus homepage & “Featured role” placement.
Priority placement in weekly AI roles newsletter.
1:1 review of your role with DevFound to calibrate for top-tier candidates.
Highlight badge on job page to increase conversion.

Best for teams hiring 1–3 critical AI roles this quarter.

Steampunk Jobs

Explore AI roles

Senior AI Evaluation Scientist

AWS Cloud Architect

AI Engineer

Data Engineer

Explore the complete AI ecosystem

Choose Your Plan

Simple pricing for serious AI hiring

Reach active AI talent

Max visibility to senior talent