LANGSMITH QA ENGINEER

Job Title: LangSmith QA Engineer
Location: 100% Remote
Type: Long-Term Contract
Compensation: Negotiable, W-2 or C2C
Work Model: Remote – offsite
Hours: 40 hours a week 
Security Clearance: No clearance but must pass background check which will include credit check 

Ideal Candidate Profile
A highly technical engineer who understands both modern software engineering and AI systems. This individual can independently assess whether an AI solution is accurate, secure, explainable, compliant, observable, and ready for production. They serve as the organization's independent quality authority for AI-powered solutions and help establish confidence in enterprise AI deployments before release to production.

Responsibilities
• Establish and execute quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions
• Develop and implement evaluation frameworks measuring accuracy, relevance, groundedness, completeness, hallucination rates, retrieval quality, recommendation quality, and user satisfaction
• Validate Retrieval-Augmented Generation (RAG) solutions utilizing Azure AI Search, Azure AI Foundry, LangChain, LangGraph, and enterprise repositories
• Perform architecture reviews and quality assessments for AI solutions deployed on Azure services such as Containers, Functions, Databricks, SQL, Cosmos DB
• Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, guardrails, retrieval pipelines, and human-in-the-loop processes
• Validate security, governance, auditability, and compliance controls, including integrations with Entra ID, RBAC, managed identities, Key Vault, and data policies
• Create production readiness checklists covering observability, monitoring, resiliency, recoverability, and operational support
• Analyze traces, logs, metrics, and telemetry data to identify quality concerns and improvement opportunities
• Collaborate with development teams for remediation of quality, security, and performance issues before deployment
• Generate certification reports, scorecards, dashboards, and executive summaries for leadership review
• Drive continuous improvements to validation methodologies, testing strategies, and evaluation frameworks
• Lead certification and production readiness reviews for internal developer portals and platform initiatives

Requirements
• Bachelor’s Degree in Computer Science, Software Engineering, Data Science, AI, or related field
• 5+ years of experience in software quality engineering, platform engineering, software development, or related roles
• 2+ years working directly with Generative AI, LLMs, RAG architectures, AI agents, or AI applications
• Strong understanding of AI evaluation methodologies including accuracy testing and hallucination detection
• Experience with Azure AI Foundry, Azure OpenAI, LangChain, LangGraph, LangSmith, or similar frameworks
• Familiarity with cloud-native architectures and Azure services (Container Apps, Functions, Databricks, SQL, Key Vault, Cosmos DB)
• Experience supporting enterprise developer platforms and self-service engineering capabilities
• Skilled in designing automated test strategies, regression testing, and certification processes
• Strong analytical, troubleshooting, and documentation skills
• Excellent stakeholder engagement and communication abilities

Preferred Qualifications
• Experience building or validating enterprise AI agents and multi-agent systems
• Knowledge of AI observability and evaluation platforms like LangSmith
• Familiarity with knowledge retrieval solutions, embeddings, and semantic search
• Background in AI governance, Responsible AI, and compliance frameworks
• Experience working in regulated industries such as healthcare, finance, or insurance
• Proficiency with Azure DevOps, GitHub, and SDLC tooling
• Knowledge of identity management concepts including Entra ID and RBAC
• Ability to produce quality scorecards, dashboards, and KPI reports