Senior Data Engineer

Job Description
Everforth ECS is seeking a highly skilled Senior Data Engineer with extensive experience designing, building, and maintaining data platforms and pipelines for large-scale, enterprise applications in support of the Department of Homeland Security. Tools necessary for excelling in this role include SQL (PostgreSQL or MySQL preferred), Elasticsearch/OpenSearch, data modeling and transformation frameworks, and DevSecOps practices in the Cloud.

This role will focus heavily on modernizing legacy data architectures and re-architecting the application's data layer, shifting from Elasticsearch to PostgreSQL or MariaDB as the system of record for transactional data. You will play a critical role in evolving scoring models, intelligence modifiers, and capability gap logic while ensuring alignment and consistency between the new relational database and Elasticsearch.

Key Responsibilities:

• Design and architect robust, scalable data models, schemas, indexes, and stored procedures in a relational database to support transactional workloads and complex business logic.
• Lead the migration of the system of record from Elasticsearch to a relational database, including data modeling, transformation, validation, and cutover strategies.
• Develop and maintain data pipelines and integration processes to keep the database and Elasticsearch aligned for search and analytics use cases.
• Implement and evolve scoring models, intelligence modifiers, and capability gap logic in the data layer to support mission-critical decision support.
• Optimize database performance, including indexing strategies, query tuning, and partitioning to support high-volume, low-latency operations.
• Define and enforce data quality, data validation, and data governance standards across legacy and modernized components.
• Collaborate with application engineers to integrate APIs and services with the underlying data models and stored procedures, ensuring seamless functionality and consistent data semantics.
• Provide technical guidance and solutions for legacy data system modernization while minimizing downtime and operational risk during migration.
• Identify and remediate data-related technical debt in older systems while transitioning to modern data engineering patterns, tools, and frameworks.
• Integrate and operate CI/CD pipelines used for building, testing, and deploying data services and schema changes in cloud environments.

Required Skills

• Ability to pass a Public Trust investigation.
• Must have US citizenship.
• Bachelor's degree in a related field.
• Minimum 15 years general work experience.
• 8+ years of experience as a Data Engineer or in a closely related role, with strong experience in legacy data system modernization.
• 5+ years of hands-on experience with relational databases including from-scratch schema design, index creation, stored procedures SQL development, performance tuning, and transaction management.
• 3+ years of experience working with Elasticsearch or OpenSearch, including index design, mappings, and data ingestion strategies.
• Demonstrated experience designing and implementing data models and logic to support scoring models, rules engines, or similar decision-support systems.
• 5+ years of experience supporting applications in production, troubleshooting complex data issues in legacy environments, and proposing effective remediation strategies.
• Strong understanding of data modeling for OLTP/transactional systems, including normalization/denormalization tradeoffs and referential integrity.
• Solid understanding of Git, version control workflows, and collaborative development practices.

Desired Skills

• Experience re-architecting systems to move the system of record between data stores (e.g., Elasticsearch to database), including coexistence, synchronization, and phased cutovers.
• Familiarity with automated testing frameworks and tools for data validation, regression testing, and schema change verification.
• Experience with the following tools: Docker, Kubernetes, Jenkins, Terraform, Ansible, OpenSearch, Elasticsearch, Gitlab.
• 5+ years of experience with relational databases and ORM technologies (e.g., JPA/Hibernate), experience with Elasticsearch or OpenSearch alignment patterns a plus.
• Experience supporting analytics, reporting, monitoring, and/or data science workloads on top of transactional systems, including exposing data needed for scoring model evolution and intelligence modifiers.
• Familiarity working in Agile / SCRUM development environments.