AI QA & Evaluation Engineer

Elastic is hiring an AI QA & Evaluation Engineer in Bangalore.

Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the results that matter. By taking advantage of all structured and unstructured data — securing and protecting private information more effectively — Elastic’s complete, cloud-based solutions for search, security, and observability help organization

Requirements

- Proficiency in Python, TypeScript, or other programming languages used in AI and test automation.

- Proven skill in designing or applying rubric-based evaluation, grading against set criteria, or building structured scoring frameworks.

- Direct experience with LLM evaluation frameworks and benchmarking tools such as LangSmith, Confident AI, etc.

- Knowledge of the GenAI stack and solutions including Retrieval Augmented Generation (RAG).

- LLMs: Azure OpenAI, Vertex AI, ChatGPT Enterprise or similar.

- High attention to detail and ability to notice subtle patterns or inconsistencies (such as data hallucinations or logic errors) that others might miss.

- Advanced written communication skills, especially for documenting nuanced observations and feedback.

- Experience with Cloud platforms (Azure, GCP, AWS).

About Elastic

See the company's official careers page for full details, then apply using the button below.