Senior Data Lake Engineer - Databricks

We are seeking a highly experienced and skilled Senior Data Lake Engineer to join our team. As the Senior Data Lake Engineer, you will play a critical role in establishing and configuring an enterprise-level Databricks solution to support our federal customer organization's data lake initiatives. This position offers a unique opportunity to work with cutting-edge technologies and shape the future of our federal customer's data infrastructure.

If you are a highly skilled and experienced Senior Data Lake Engineer with expertise in Databricks and passion for building scalable and secure data lake solutions, we would like to hear from you.

Responsibilities:

• Lead the design, implementation, and configuration of an enterprise Data Lake solution utilizing Databricks, ensuring scalability, reliability, and optimal performance.
• Collaborate with cross-functional teams to gather requirements, understand data integration needs, and define data lake architecture and governance policies.
• Establish and configure Databricks workspaces, clusters, and storage components, optimizing the solution for efficient data processing, query performance, and data governance.
• Design and implement data ingestion pipelines to efficiently extract, transform, and load data from various sources into the data lake using Databricks tools and services.
• Develop and maintain data lake security frameworks, including access controls, encryption solutions, and data masking techniques to protect sensitive data.
• Collaborate with data engineers and data scientists to optimize data pipelines, develop data transformations, and ensure data quality and integrity.
• Monitor and tune Databricks clusters and workloads to ensure performance, reliability, and cost optimization, utilizing automated scaling and resource management techniques.
• Implement best practices for data governance, data cataloging, metadata management, and data lineage within Databricks, adhering to regulatory and compliance requirements.
• Collaborate with infrastructure teams to ensure data lake infrastructure meets scalability and availability requirements, leveraging Databricks cluster management and AWS/Azure services.
• Develop and maintain documentation and guidelines related to the Databricks solution, including architecture diagrams, standards, and processes.
• Stay up to date with the latest advancements in Databricks, big data technologies, and cloud platforms, continuously evaluating and implementing new features and capabilities.
• Provide technical guidance and mentorship to junior data engineers, promoting best practices and fostering a culture of continuous learning and growth.
• Collaborate with stakeholders to understand their data analytics and reporting needs and develop scalable data models and data transformation processes to support these requirements.
• Support data lake-related incident resolutions, troubleshooting data quality issues, performance bottlenecks, and other data-related challenges.
• Collaborate with data governance and compliance teams to ensure data privacy, security, and compliance guidelines are adhered to within the data lake solution.
• Participate in the evaluation and selection of new tools, technologies, and services to enhance the data lake infrastructure.

Minimum Qualifications

• Bachelor's degree in computer science, information technology, or a related field. Equivalent experience will also be considered.
• Proven experience in building and configuring enterprise-level data lake solutions using Databricks in an AWS or Azure environment.
• In-depth knowledge of Databricks architecture, including workspaces, clusters, storage, notebook development, and automation capabilities.
• Strong expertise in designing and implementing data ingestion pipelines, data transformations, and data quality processes using Databricks.
• Experience with big data technologies such as Apache Spark, Apache Hive, Delta Lake, and Hadoop.
• Solid understanding of data governance principles, data modeling, data cataloging, and metadata management.
• Hands-on experience with cloud platforms like AWS or Azure, including relevant services like S3, EMR, Glue, Data Factory, etc.
• Proficiency in SQL and one or more programming languages (Python, Scala, or Java) for data manipulation and transformation.
• Knowledge of data security and privacy best practices, including data access controls, encryption, and data masking techniques.
• Strong problem-solving and analytical skills, with the ability to identify and resolve complex data-related issues.
• Excellent interpersonal and communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.
• Experience in a senior or lead role, providing technical guidance and mentorship to junior team members.
• Relevant certifications such as Databricks Certified Developer or Databricks Certified Professional are highly desirable.
• Eligibility/Clearance Requirements : Must be able to provide proof of U.S. Citizenship.

Desired Qualifications

Clearance Preference :

• Active DHS/CISA suitability - 1st priority
• Any DHS badge + DoD Top Secret - 2nd choice
• DoD Top Secret + willingness to obtain DHS/CISA suitability - 3rd choice (it can take 10-60 days to obtain suitability - work can only begin once suitability is fully adjudicated).

About Sev1Tech LLC

Founded in 2010, Sev1Tech provides IT, engineering, and program management solutions delivery. Sev1Tech focuses on providing program and IT support services to critical missions across Federal and Commercial Clients. Our Mission is to Build better companies. Enable better government. Protect our nation. Build better humans across the country.

• Join the Sev1Tech family where you can achieve great accomplishments while fostering a satisfying and rewarding career progression. Please apply directly through the website at: https://careers-sev1tech.icims.com/ #joinSev1tech
• For any additional questions or to submit any referrals, please contact: Caitlin.maupin@sev1tech.com
• Sev1Tech is an Equal Opportunity and Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.