Role overview
- AI Model Testing & Validation: Design and implement model validation frameworks: accuracy, precision, recall, F1 across clinical subgroups (age, hearing loss severity, device type).
- Write regression tests for model updates — detect silent accuracy degradation before any deployment to production.
- Validate model outputs: Design adversarial test cases: edge cases, out-of-distribution inputs, boundary conditions, and clinically implausible inputs.
- AI API & Integration Testing: Design and maintain API test suites for AI feature endpoints consumed by the PMS frontend and microservices.
- Write contract tests between AI services and consuming services — prevent integration breakage when models are updated or retrained.
- Test latency, throughput, and graceful degradation under load (AI inference endpoints have stricter SLAs than standard CRUD APIs).
- Validate error handling: model confidence thresholds, fallback behaviour when models are unavailable or return low-confidence outputs.
- Collaborate with the AQA on shared Playwright E2E coverage for AI-integrated UI flows.
- ISO 42001 & Regulatory Compliance Testing: Design testing evidence for ISO 42001 AIM - Straceability from AI system requirements to test cases to results, supporting audit and certification.
- Observability & Production Monitoring: Design and implement model monitoring pipelines - track accuracy, confidence distribution, and prediction drift in production against baseline.
- Contribute to post-incident reviews when AI features cause unexpected clinical workflow impacts or regulatory flags.
- Feed production monitoring findings back into the regression test suite to prevent recurrence and improve model robustness
What we're looking for
- Familiarity with Python or willingness to become comfortable with it over time.
- Statistical testing methods: hypothesis testing, A/B evaluation, bootstrap confidence intervals.
- LLM evaluation and testing patterns (prompt regression testing, output consistency, hallucination detection).
- AKS/Kubernetes, Azure Service Bus, Azure SQL, Azure AI Foundry
Interview process
We strive to make our hiring process smooth and transparent to find the perfect match for both sides. Steps may differ depending on the role, but here’s what to expect:
- Initial Interview: If your background fits the role, we’ll invite you for an interview with a Talent Acquisition Specialist.
- Technical Interview: Depending on the position, you may complete a technical assessment or test task.
- Final Decision: After all steps, we’ll get back to you with the result and next steps.
Tags & focus areas
Used for matching and alerts on DevFound Remote Ai Machine Learning