Today
Secret
Mid Level Career (5+ yrs experience)
IT - Database
Senior Data Pipeline Engineer
REMOTE
This position requires an ACTIVE security clearance Secret or higher.
This position can be performed remote from anywhere, but may require up to 15% travel.
As a Senior Data Pipeline Engineer, you will play a crucial role in designing, building, and maintaining robust data pipelines for our customers. Your expertise will drive efficient collection, storage, processing, and transformation of large-scale data sets in support of the U.S. Army’s energy and water use metering program. The total solution is considered a system-of-systems with developed applications integrating with the Army enterprise cloud and data environment. You will work closely with cross-functional teams to ensure seamless integration and optimal performance.
Responsibilities:
Pipeline Development:
• Design, develop, and optimize end-to-end data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable pipeline solutions.
• Implement best practices for data integration, ensuring high performance, reliability, and scalability.
Data Transformation and Quality:
• Transform raw data into usable formats, ensuring data quality, consistency, and accuracy.
• Handle data validation, cleansing, and error handling to maintain data integrity.
• Monitor and proactively maintain data pipelines to ensure high service availability.
Performance Optimization:
• Continuously improve pipeline performance by identifying bottlenecks and implementing optimizations.
• Work with cloud-based technologies (e.g., AWS, GCP, Azure) to enhance scalability and efficiency.
Collaboration and Leadership:
• Partner with Data Scientists, Analysts, and other stakeholders to understand their data needs.
• Lead discussions on system enhancements, process improvements, and data governance.
• Mentor junior engineers and contribute to the growth of the data engineering team.
Requirements
REQUIRED QUALIFICATIONS:
• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering, with a focus on building and maintaining data pipelines.
• 1-2 years experience with building NiFI data flows or similar for Kafka and Hadoop-based NoSQL databases.
• Proficiency in ETL tools, SQL, and scripting languages (Python, Scala, etc.).
• Experience with Data Catalog and Accumulo indexes for information retrieval and discovery.
• Experience with API-led design.
• Experience with ELK stack a plus.
• Experience with cloud-based data platforms (e.g., AWS S3, Redshift, Google BigQuery).
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration abilities.
• Security+ Certification
• Active US Government Clearance at Secret level or higher
• Effective written and verbal communications skills for collaboration with both customers and fellow team members.
• Ability to sit for extended periods of time.
• Ability to regularly lift at least 25 pounds.
• Ability to commute to the designated onsite work location as required.
REMOTE
This position requires an ACTIVE security clearance Secret or higher.
This position can be performed remote from anywhere, but may require up to 15% travel.
As a Senior Data Pipeline Engineer, you will play a crucial role in designing, building, and maintaining robust data pipelines for our customers. Your expertise will drive efficient collection, storage, processing, and transformation of large-scale data sets in support of the U.S. Army’s energy and water use metering program. The total solution is considered a system-of-systems with developed applications integrating with the Army enterprise cloud and data environment. You will work closely with cross-functional teams to ensure seamless integration and optimal performance.
Responsibilities:
Pipeline Development:
• Design, develop, and optimize end-to-end data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable pipeline solutions.
• Implement best practices for data integration, ensuring high performance, reliability, and scalability.
Data Transformation and Quality:
• Transform raw data into usable formats, ensuring data quality, consistency, and accuracy.
• Handle data validation, cleansing, and error handling to maintain data integrity.
• Monitor and proactively maintain data pipelines to ensure high service availability.
Performance Optimization:
• Continuously improve pipeline performance by identifying bottlenecks and implementing optimizations.
• Work with cloud-based technologies (e.g., AWS, GCP, Azure) to enhance scalability and efficiency.
Collaboration and Leadership:
• Partner with Data Scientists, Analysts, and other stakeholders to understand their data needs.
• Lead discussions on system enhancements, process improvements, and data governance.
• Mentor junior engineers and contribute to the growth of the data engineering team.
Requirements
REQUIRED QUALIFICATIONS:
• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering, with a focus on building and maintaining data pipelines.
• 1-2 years experience with building NiFI data flows or similar for Kafka and Hadoop-based NoSQL databases.
• Proficiency in ETL tools, SQL, and scripting languages (Python, Scala, etc.).
• Experience with Data Catalog and Accumulo indexes for information retrieval and discovery.
• Experience with API-led design.
• Experience with ELK stack a plus.
• Experience with cloud-based data platforms (e.g., AWS S3, Redshift, Google BigQuery).
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration abilities.
• Security+ Certification
• Active US Government Clearance at Secret level or higher
• Effective written and verbal communications skills for collaboration with both customers and fellow team members.
• Ability to sit for extended periods of time.
• Ability to regularly lift at least 25 pounds.
• Ability to commute to the designated onsite work location as required.
group id: 10430885
