Senior AI Engineer - LLM Evaluation

Autodesk is hiring a Senior AI Engineer - LLM Evaluation in Bengaluru.

Job Requisition ID # 26WD98447 Position Overview If you love building real Artificial Intelligence (AI) systems that solve real customer problems and are passionate about measuring, improving, and scaling AI quality, this role is for you. The Applied AI team within Autodesk's Architecture, Engineering, and Construction (AEC) organization builds cloud-native AI solutions that make industry workflows smarter, faster, and more connected.

Responsibilities

• Lead AI evaluation and quality strategy for Generative AI and Machine Learning (ML)-powered systems
• Design and implement scalable evaluation frameworks for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, agentic workflows, and traditional Machine Learning (ML) models
• Fine-tune, evaluate, benchmark, and deploy Large Language Models (LLMs) in production environments
• Implement evaluation and observability standards, including regression testing, monitoring, tracing, feedback loops, and quality measurement for AI systems
• Define and track AI quality metrics, including accuracy, robustness, bias, drift, grounding, latency, safety, and hallucination rates
• Build automated evaluation pipelines and integrate them into Continuous Integration and Continuous Deployment (CI/CD) workflows for continuous model validation
• Evaluate and adopt AI testing frameworks, benchmark methodologies, and emerging evaluation tools
• Build scalable Retrieval-Augmented Generation (RAG) pipelines using structured and unstructured enterprise data sources
• Integrate AI agents with Autodesk platforms and enterprise services using Model Context Protocol (MCP)-based integrations or equivalent architectures
• Partner with Product Managers, Architects, Software Engineers, Experience Designers, and Data Scientists to deliver AI-powered experiences
• Ensure AI systems align with Responsible AI principles and enterprise governance requirements
• Contribute to reusable AI platform capabilities, Software Development Kits (SDKs), orchestration patterns, and engineering best practices
• Leverage AI coding assistants to accelerate feature delivery while maintaining rigorous validation through testing, Continuous Integration and Continuous Deployment (CI/CD), code reviews, and security best practices

Requirements

• Bachelor's or Master's degree in Computer Science, Machine Learning, Data Science, Engineering, or equivalent practical experience
• 5+ years of hands-on experience delivering Artificial Intelligence (AI) and Machine Learning (ML)-powered systems into production
• Experience designing evaluation frameworks, automated testing solutions, benchmarking systems, or model validation pipelines
• Strong understanding of Large Language Model (LLM) evaluation, model quality measurement, and AI observability concepts
• Strong proficiency in Python with experience building automation frameworks and testing infrastructure
• Experience deploying and operating AI applications in production, including monitoring, observability, and continuous improvement
• Experience building cloud-native applications on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
• Excellent technical communication and collaboration skills
• Demonstrated experience using AI coding tools to accelerate the delivery of production systems

Preferred qualifications

• Hands-on experience with AI evaluation frameworks such as LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals, or similar tools
• Experience evaluating Retrieval-Augmented Generation (RAG) systems, agentic workflows, AI assistants, and prompt-based applications
• Familiarity with Responsible AI, AI governance, model risk assessment, and AI observability platforms
• Experience deploying cloud-native AI solutions on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
• Familiarity with deep learning architectures such as Transformers and modern Machine Learning (ML) frameworks, including PyTorch, Lightning, Ray, or equivalent technologies
• Experience with enterprise knowledge systems, semantic search, vector search, or contextual retrieval systems
• Experience with emerging agentic AI frameworks such as LangGraph, CrewAI, OpenAI Software Development Kits (SDKs), Claude Software Development Kits (SDKs), Model Context Protocol (MCP), Agent-to-Agent (A2A), or equivalent technologies
• Contributions to open-source AI projects, technical blogs, conference presentations, or published research are an advantage

The Ideal Candidate

• Demonstrates a strong understanding of Artificial Intelligence (AI), Machine Learning (ML), and Generative AI concepts and applies industry best practices to solve complex technical challenges
• Uses analytical thinking, sound judgment, and data-driven decision-making to evaluate model performance and recommend effective solutions
• Connects technical implementation with broader product, customer, and business objectives
• Applies creativity and innovation to improve AI quality, evaluation methodologies, and engineering practices
• Leads projects or key workstreams while contributing to the successful delivery of broader initiatives
• Takes ownership of improving engineering processes, quality standards, and operational excellence
• Builds strong working relationships with senior technical leaders, peers, and cross-functional stakeholders

About Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made. We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and define