Software Engineer – Distributed Systems & Apache Kafka

Candidates must be willing to work in SCIF daily or as needed. The anticipated end for this work is the end of July, with expectation of mod/additional funding through 2027.

Responsibilities

• Design, build, and maintain highly scalable Apache Kafka streaming systems to support near real-time data ingestion and processing across distributed environments
• Define and develop comprehensive technical onboarding requirements for new data types, ensuring seamless integration and compatibility with existing streaming infrastructure
• Develop and maintain robust metrics reporting for Kafka services, including end-to-end latency, throughput, and operational status to drive performance visibility and proactive issue resolution

Requirements

• TS/SCI CI Poly Clearance with VA Customer eligibility and SBU/SECNet/COE accounts
• DoD 8140 Compliance
• 5+ years of professional software engineering experience with cloud-first development approach
• 5+ years of hands-on experience with AWS CDK for Infrastructure as Code
• 5+ years of advanced Python development experience
• 4+ years of data engineering experience
• 3+ years building complex, reusable CDK constructs and stacks
• Proficiency in async programming and performance optimization* 3+ years implementing multi-environment deployment strategies using CDK
• 3+ years with AWS cloud-native services:
  
  
  
  • Advanced knowledge of Lambda, API Gateway, EventBridge, SQS, SNS
  
  
  • Experience with data services: RDS, DynamoDB, Redshift, S3, Athena
  
  
  • Container orchestration with ECS, Fargate, or EKS
  
  
  
  


• API design and versioning (preferably with Apache Kafka)


• ETL/ELT pipeline development using AWS services


• Experience with data lakes, data warehousing, and analytics platforms
  
  
  
  • Real-time data processing and streaming architectures
  
  
  
  


• Strong problem-solving abilities with proven track record of resolving complex technical challenges

Desired Skills

• 3+ years of experience designing and maintaining distributed streaming systems, with hands-on Apache Kafka expertise


• Demonstrated ability to develop and document technical onboarding or integration requirements for complex data pipelines


• Experience building metrics dashboards and monitoring systems to track latency, throughput, and operational health of streaming services


• Familiarity with cloud infrastructure and DevOps practices for deploying and scaling data streaming platforms

About Us

For more than 20 years, NewGen Technologies has solved our clients’ toughest IT challenges with integrity, security, and outstanding service by delivering both technology and talent. We have helped secure borders, have used artificial intelligence (AI) to fight terror, aided the identification of criminals, and have helped to prevent crime through the introduction of biometrics. Our team of Highly Cleared Specialists have hard-to-find skills and expertise in a wide spectrum of technologies to provide solutions that transform business processes and solve problems of national significance. #CJ