Staff Software Development Test Engineer - AI Evaluation

Tekion is hiring a Staff Software Development Test Engineer - AI Evaluation in Bangalore HQ.

Responsibilities

• Develop a deep understanding of Tekion’s AI agents, ML models, and the quality dimensions that matter for each business domain.

• Design, enhance, and own Tekion’s AI evaluation infrastructure as a shared capability used across ML teams.

• Create, curate, and maintain evaluation datasets (evals) and golden/ground-truth sets across use cases and domains.

• Define quality metrics for AI outputs — accuracy, relevance, faithfulness/groundedness, consistency, safety, and task success.

• Build automated scoring pipelines, including LLM-as-judge, rubric-based, and referencebased evaluation methods.

• Validate user intents and measure response accuracy and consistency for AI-powered capabilities such as the Analytics Agent.

• Identify hallucinations, unsafe or biased outputs, and edge cases; design targeted eval suites to catch them.

• Build both offline evaluation (pre-release benchmarking) and online evaluation (production quality monitoring, A/B, drift detection).

• Establish evaluation gates in CI/CD so model, prompt, or data changes are quality-checkedbefore release.

• Develop dashboards and reporting that make AI quality visible and actionable for ML and product teams.

• Use AI/LLMs to scale evaluation — automated judges, synthetic data generation, and eval \tooling

• Champion evaluation and responsible-AI quality best practices across the organization.

Requirements

• 5–8 years in SDET, quality engineering, ML engineering, or data science, with hands-on experience building evaluation or measurement systems — or a strong SDET background with deep LLM/ML fluency.

• Strong programming skills in Python, with the ability to build robust, reusable evaluation pipelines and tooling.

• Deep understanding of ML/LLM evaluation, benchmark design, and the pitfalls of evaluating non-deterministic systems.

• Hands-on experience with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.

• Solid grasp of LLM/agent concepts — prompting, RAG, embeddings, tool use — and generative failure modes (hallucination, drift, prompt sensitivity, bias).

• Experience designing and curating datasets, including labeling/annotation strategy and data quality.

• Strong statistical intuition for interpreting eval results and significance.

• Excellent communication skills to translate quality signals into decisions for ML and product teams.

Preferred qualifications

• Experience with eval frameworks/tools such as Ragas, DeepEval, LangSmith, TruLens, Promptfoo, HELM, or provider eval suites.
• Experience building online evaluation, guardrails, or production model monitoring.
• Familiarity with responsible AI / safety evaluation and red-teaming.
• Experience with experiment tracking (MLflow, Weights & Biases) and A/B testing.
• Prior work standing up evaluation as a platform capability for multiple teams

Tekion is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, victim of violence or having a family member who is a victim of violence, the intersectionality of two or more protected categories, or other applicable legally protected characteristics.

For more information on our privacy practices, please refer to our Applicant Privacy Notice here https://tekion.com/legal/privacy/applicant-and-candidate-privacy-notice.

About Tekion

Positively disrupting an industry that has not seen any innovation in over 50 years, Tekion has challenged the paradigm with the first and fastest cloud-native automotive platform that includes the revolutionary Automotive Retail Cloud (ARC) for retailers, Automotive Enterprise Cloud (AEC) for manufacturers and other large automotive enterprises and Automotive Partner Cloud (APC) for technology and industry partners. Tekion connects the entire spectrum of the automotive retail ecosystem through