VLM Engineer

Responsibilities

• Design and execute fine-tuning pipelines for Vision-Language Models (VLMs) on domain-specific imagery datasets, including data preprocessing, training orchestration, and hyperparameter optimization


• Develop and implement evaluation frameworks for multimodal model performance, including task-specific metrics for image understanding, visual question answering, and spatial reasoning


• Build scalable training infrastructure on AWS (SageMaker, EC2 GPU instances) for distributed fine-tuning of large multimodal models 


• Engineer data pipelines for curating, annotating, and transforming geospatial imagery datasets into model-ready formats for supervised and instruction-tuning workflows


• Collaborate with applies scientists and solutions architects to iterate on model architectures, adapter strategies (LoRA/QLoRA), and inference optimization techniques

Requirements

• TS/SCI CI Poly Clearance with current NGA eligibility and SBU/SecNet/COE accounts


• Must be willing to work in SCIF daily or as needed


• 5+ years of professional machine learning engineering experience with a focus on deep learning


• 4+ years of advanced Python development for ML workloads


• 3+ years of experience with computer vision or multimodal models


• 3+ years of experience with AWS ML infrastructure


• SageMaker Training jobs, Processing jobs, and endpoint deployment


• GPU instance selection, multi-node training, and cost optimization on EC2  (P4/P5/G5/G6e)


• 2+ years of experience building ML evaluation pipelines


• Automated benchmarking, metric computation, and result analysis


• Experience with both quantitative metrics and qualitative/human evaluation approaches


• 1+ years of hands-on experience fine-tuning large foundation models (LLMs or VLMs)


• Experience with parameter-efficient fine-tuning methods (LoRA, QLoRA adapters)


• Familiarity with supervised fine-tuning, instruction tuning, and RLHF/DPO alignment techniques


• Strong proficiency with PyTorch and the HuggingFace ecosystem (Transformers, PEFT, Datasets, Accelerate)


• Experience with distributed training frameworks (DeepSpeed, FSDP, or Megatron)


• Understanding of vision transformer architectures (ViT, CLIP, LLaVA-family models, or similar)


• Experience processing and augmenting image datasets at scale


• Strong software engineering fundamentals (version control, CI/CD for ML workflows, testing)

Desired Skills

• 2+ years of experience with geospatial or remote sensing imagery 


• 2+ years of experience with Authority to Operate (ATO) processes in government environments


• Familiarity with electro-optical and SAR satellite imagery formats and characteristics


• Understanding of geospatial metadata, coordinate systems, and imagery preprocessing 


• Experience with model quanitzation and inference optimization (vLLM, TensorRT, ONNX)


• Experience with MLOps and experiment tracking tools (MLflow, Weights & Biases, SageMaker Experiements)


• Familiarity with data annotation platforms and active learning workflows for imagery


• Experience with containerized ML workflows (Docker, ECR, ES/EKS)


• Implementation of NIST 800-53 controls and security compliance for ML systems


• Experience deploying models in air-gapped or disconnected environments


• Familiarity with multimodal evaluation benchmarks (MMMU, MMBench, GQA, or domain-specific equivalents)


• Publications or demonstrated contributions in computer vision, VLMs, or multimodal AI


• Experience with synthetic data generation for training data augmentation 

About Us

For more than 20 years, NewGen Technologies has solved our clients’ toughest IT challenges with integrity, security, and outstanding service by delivering both technology and talent. We have helped secure borders, have used artificial intelligence (AI) to fight terror, aided the identification of criminals, and have helped to prevent crime through the introduction of biometrics. Our team of Highly Cleared Specialists have hard-to-find skills and expertise in a wide spectrum of technologies to provide solutions that transform business processes and solve problems of national significance. #CJ