Machine Learning Engineer (AI & LLM Models)

Machine Learning Engineer / AI & LLM Specialist (SME)

Location: McLean, VA​
Key Requirement: Hands-on experience testing and evaluating Retrieval-Augmented
Generation (RAG) systems

Overview:

Square Peg Technologies is seeking a Machine Learning Engineer SME to support an
Intelligence Community customer in McLean, VA. This role focuses on developing, fine tuning,
testing, and evaluating AI/ML models, particularly large language models (LLMs) and
Retrieval-Augmented Generation (RAG) systems. The ideal candidate has strong Python and
deep learning experience with expertise in GPU accelerated computing, model benchmarking,
and AI system performance evaluation.

Key Responsibilities:

• Develop, train, and fine tune LLMs and neural networks using custom datasets.

• Test and evaluate RAG systems to assess model performance and effectiveness.

• Design and execute experiments to benchmark LLM performance against established
evaluation criteria.

• Develop and evaluate machine learning models using Python and deep learning frameworks.

• Leverage GPUs to accelerate model training, inference, and testing.

• Develop data preprocessing workflows, select appropriate ML libraries, and optimize model
performance.

• Test, validate, and compare AI/ML models for specific use cases.

• Develop SQL queries to extract, transform, and analyze data.

• Use GitHub and Jenkins to support software development and testing workflows.

• Document experimental results, model performance, and technical recommendations.

Required Qualifications:

• Hands-on experience testing and evaluating RAG systems.

• Experience fine tuning LLMs or neural networks using custom datasets.

• Experience developing machine learning models and ensembles, including preprocessing,
model selection, and performance evaluation.
,• Strong programming experience with Python or R.

• Experience with deep learning frameworks such as PyTorch, TensorFlow, or Keras.

• Proficiency with SQL, including common table expressions, set operations, aggregate
functions, and nested subqueries.

• Experience with GitHub and Jenkins.

• Hands-on experience leveraging GPUs for accelerated computing.

• Experience testing, evaluating, and validating AI systems, particularly LLM based
systems.

• Experience designing and executing experiments to benchmark LLM performance.

Desired Qualifications:

• Experience with Hugging Face Transformers and Hugging Face Hub.

• Experience developing benchmarks to measure LLM performance.

• Experience with cloud computing development and architecture.

• Experience with Flask or similar web development frameworks.

• Experience developing text classification and topic modeling solutions using Scikit Learn or
deep learning models.

• Experience developing semantic search applications.

• Experience creating dashboards and visualizations using Tableau.

• Ability to communicate model methodologies, technical decisions, and experimental results.

• Experience with verification and validation test benches.

• Knowledge of Explainable AI (XAI) techniques.

• Experience with ONNX (Open Neural Network Exchange).