AI Evaluation Specialist

AI Evaluation Specialist
Remote

Blu Omega is seeking an AI Evaluation Specialist to support the assessment, testing, and adoption of enterprise AI tools and capabilities for mission-critical federal programs. This role is responsible for evaluating generative AI products, models, and AI-enabled applications within an enterprise technology environment.

Program Overview

This program supports federal efforts to evaluate and adopt enterprise AI capabilities.

Key Details

• Location: Remote
• Clearance / Suitability: Public Trust eligibility required.

Responsibilities

• Design and execute hands-on evaluations of generative AI tools, enterprise platforms, and LLM-based applications, including OpenAI and specialized life sciences AI solutions.
• Develop standardized evaluation frameworks, test cases, scoring rubrics, benchmarks, and acceptance criteria to assess AI quality and performance.
• Evaluate AI solutions for accuracy, relevance, consistency, reasoning, reliability, usability, latency, and overall effectiveness.
• Identify hallucinations, model limitations, security vulnerabilities, privacy concerns, and other potential AI risks.
• Conduct comparative assessments and proof-of-concept testing across AI models, platforms, configurations, and vendors.
• Evaluate retrieval-augmented generation (RAG), AI agents, enterprise search, APIs, connectors, and Model Context Protocol (MCP)-enabled workflows.
• Collaborate with technical and business stakeholders to assess AI capabilities, document findings, and recommend improvements.
• Develop evaluation reports, scorecards, and performance metrics to support AI adoption and governance decisions.
• Stay current with emerging AI technologies, evaluation methodologies, and responsible AI practices.

Required Qualifications

• 5+ years of experience in model evaluation, quality assurance, or related technical evaluation roles, including hands-on assessment of AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring methodologies, benchmarks, and measurable acceptance criteria.
• Hands-on experience testing generative AI tools and LLMs, including prompt engineering, output analysis, and identifying hallucinations or model failure modes.
• Experience evaluating AI systems for accuracy, reliability, safety, bias, privacy, security, usability, and performance.
• Understanding of generative AI architectures and capabilities, including LLMs, RAG, AI agents, APIs, and enterprise integrations.
• Strong analytical, problem-solving, and documentation skills with the ability to communicate technical findings to diverse stakeholders.
• Bachelor's degree.

Preferred Qualifications

• Experience evaluating OpenAI products, Rosalind, or comparable enterprise and life sciences AI platforms.
• Experience with automated LLM evaluation frameworks, LLM-as-a-judge, human-in-the-loop testing, red teaming, or adversarial testing.
• Experience evaluating AI agents, RAG systems, MCP integrations, and multi-step AI workflows.
• Proficiency with Python, SQL, notebooks, or analytics tools for analyzing evaluation results and performance metrics.
• Experience developing dashboards and reporting solutions to track AI quality, risks, and evaluation trends.
• Industry certifications in AI, machine learning, data science, cloud technologies, cybersecurity, or responsible AI.
• Master's degree.

Compensation

Salary Range: $125,000.00 - $135,000.00

Final compensation is based on technical skills, experience, education, certifications, and clearance.

Blu Omega Benefits & Perks

• Medical, Dental, and Vision coverage through national providers
• 401(k) with company match (eligible after 6 months; vesting applies)
• Company-paid Life and AD&D insurance with additional voluntary options
• Short-term disability (company-paid) and long-term disability options
• Employee Assistance Program (EAP) with 24/7 confidential support and mental health resources
• Telehealth and virtual care options available through medical plans
• Pet insurance, legal services, and identity theft protection options
• Paid Time Off (PTO) for eligible employees
• Paid federal holidays for salaried employees
• Access to wellness programs, discounts, and lifestyle benefits

Benefits eligibility and offerings may vary based on role, employment status, and program requirements.

Company Overview

Blu Omega is a Woman-Owned Small Business (WOSB) delivering technology and cybersecurity solutions to federal agencies and enterprise clients nationwide. Headquartered in Ashburn, VA, we support mission-critical programs across civilian and defense sectors, including health, national security, and regulatory environments. We partner with government agencies and large integrators to provide expertise in cybersecurity operations, cloud and infrastructure modernization, data and analytics, and enterprise IT support.

Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, or disability.

#CJ

#LI-Remote