Role overview
Summary
Andersen is hiring a Senior AI Engineer (Observability) to build scalable AI infrastructure and observability solutions for a cloud platform, ensuring reliable performance and intelligent system monitoring.
The customer is an international company delivering professional and technology-enabled solutions that support effective collaboration, structured communication, and operational efficiency for organizations. It operates in a fast-growing environment, focusing on scalability, security, and continuous improvement while developing digital platforms used by diverse clients worldwide.
The project is focused on enhancing a secure, cloud-based board management platform with intuitive meeting tools, real-time collaboration, and advanced analytics. It also includes building and maintaining scalable AI infrastructure, orchestration patterns, and observability to ensure reliable and intelligent platform performance.
What we're looking for
- Translating product requirements into AI workflows and quality criteria.
- Designing, implementing, and optimizing prompts using LangFuse.
- Building evaluation datasets from real customer scenarios.
- Establishing quality gates and measurement systems (precision, recall, accuracy).
- Setting up and maintaining LangFuse for observability and prompt experimentation.
- Tuning workflows based on production telemetry and evaluation results.
- Partnering with Product and QE on acceptance criteria and test strategy.
- Experience in software engineering for 5+ years, including delivering production systems.
- Bachelor's degree in computer science, Engineering, IT, or equivalent experience.
- Experience with LLM-powered features or AI pipelines (instrumentation, evaluation, or monitoring).
- Experience with LLMOps and observability tools (e.g., Langfuse, Arize, W&B, LangSmith).
- Understanding RAG, prompt engineering, vector search, and orchestration frameworks (e.g., Semantic Kernel, LangChain).
- Experience in designing LLM evaluations (datasets, scoring, LLM-as-judge, regression testing).
- Proficiency in Python and/or C#/.NET, including working with production codebases.
- Understanding observability fundamentals (tracing, logging, metrics, alerting) in cloud environments (e.g., OpenTelemetry, Azure Monitor).
- Experience with CI/CD and Git workflows, Azure DevOps preferred.
- Understanding LLM cost management (token tracking, budgeting, optimization).
- Experience working in regulated environments (e.g., SOC 2, ISO 27001).
- Experience with AI-assisted development tools (e.g., Claude Code, PlayerZero).
- Strong communication skills, including translating insights and educating teams on AI quality.
- Level of English – from Intermediate+ and above.
Desired skills
- Experience with A/B testing or online experimentation frameworks applied to AI features.
- Familiarity with OpenTelemetry and its emerging application to LLM tracing (OpenLLMetry or similar).
- Experience monitoring guardrail and content safety systems — including tracking filter effectiveness, policy enforcement, and safety-related anomalies as a distinct observability concern.
- Experience with semantic similarity metrics, embedding models, and vector database platforms (e.g., Azure AI Search, Pinecone, Qdrant).
- Background in QA engineering, data engineering, or ML engineering that informs a rigorous approach to quality measurement.
- Experience extending observability or platform practices across multiple product lines or teams.
Reasons to join us
- Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
- The opportunity to change the project and/or develop expertise in an interesting business domain.
- Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
- Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
- The opportunity to earn an additional up to 1,000 USD per month by participating in the company's activities.
- Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
- Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
- Certification compensation (AWS, PMP, etc).
- Referral program.
- English courses.
- Private health insurance and compensation for sports activities.
Join us!
Locations
Albania, Austria, Azerbaijan, Belarus, Belgium, Bosnia and Herzegovina, Bosnia and Herzegovina, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Georgia, Germany, Hungary, Italy, Kazakhstan, Kosovo, Kyrgyzstan, Latvia, Lithuania, Luxembourg, Montenegro, Norway, Poland, Portugal, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, The Netherlands, The Republic of Moldova, Turkmenistan
Show more