4351 Splunk Cloud Engineer

OVERVIEW:

The IA&E Engineer provides advanced cybersecurity engineering and infrastructure expertise in support of Treasury's enterprise-wide SOC operations under TESIEMS. This position is responsible for designing, integrating, and maintaining secure compute, storage, and virtualization environments across both on-premises data centers and cloud environments (AWS). Additionally, this role configures and troubleshoots Splunk platform components, focusing on resolving connectivity and performance issues across the Splunk environment. Ensure platform availability, data integrity, and reliability of log flow from multiple sources. The engineer applies deep technical knowledge of Linux systems, virtual machines, and enterprise infrastructure to optimize SOC monitoring, strengthen Treasury's cybersecurity posture, and ensure resilient, scalable operations across the Department.

GENERAL DUTIES:

• Primary responsibilities include design, configure, and maintain low-level Splunk platform components including forwarders, indexers, search heads, and deployment servers.
• Troubleshoot system outages, downtimes, and performance degradation.
• Diagnose and resolve issues with unresponsive or misconfigured hosts and forwarders. Investigate and remediate syslog ingestion issues, including network connectivity, parsing, and logging issues.
• Support integration and troubleshooting of security and email gateway logs, including IronPort data sources.
• Configure, maintain, and optimize Splunk configuration files including props.conf, inputs.conf, transforms.conf, and related configurations.
• Analyze logs to identify root causes of ingestion delays, dropped events, or parsing errors.
• Monitor data pipelines to ensure consistent log ingestion, normalization, and indexing.
• Collaborate with system administrators, network teams, and security teams to resolve cross-platform issues.
• Document troubleshooting procedures, configuration changes, and operational best practices.
• Develop dashboards and Splunk Health Checks to check intermediate forwarder connectivity, search head availability, disk utilization, and license usage.
• Configure and manage server classes to control forwarder configurations, app distribution, and deployment across the environment.
• Manage and troubleshoot Cluster Manager controls, including index replication, bucket fixing, peer status, and cluster health.
• Configure, deploy, and support intermediate forwarders, including load balancing, filtering, routing, and data normalization.
• Coordinate planned maintenance, upgrades, and configuration changes ensuring minimal disruption to data ingestion and search availability.
• Develop and maintain operational runbooks, health check procedures, and escalation workflows.

REQUIRED QUALIFICATIONS:

• Seven (7) years of experience. Three additional years of experience in lieu of degree.
• Deep Splunk technical knowledge and proficiency.
• Problem-solving complex Splunk data flow and configuration technical issues.
• Proven experience supporting Splunk in a production environment with a focus on low-level operations and troubleshooting.
• Strong understanding of Splunk data ingestion pipelines, forwarders, indexing processes, and search performance.
• Hands-on experience configuring and troubleshooting props.conf, inputs.conf, and transforms.conf.
• Strong Linux/Unix system administration skills, including log analysis, process monitoring, and network troubleshooting.
• Excellent problem-solving skills and strong communication skills for coordinating with technical teams.
• Strong analytical and critical thinking skills.
• Understanding of impact assessment on end-products or solutions.
• Broad technical understanding of related cybersecurity specialty areas.
• Ability to develop and implement technical solutions independently.
• Familiarity with incident detection, response, and security event management. Proficiency in tools such as SIEMs (e.g., Splunk), IDS/IPS, endpoint detection, and scripting languages.
• Familiarity with NIST SP 800-53, FISMA, and risk management frameworks.
• Experience with scripting (e.g., Python, Bash) and log data analysis.

DESIRED QUALIFICATIONS:

• Bachelor's degree from an accredited institute in an area applicable to the position in Cybersecurity, Computer Science, Information Systems, or a related discipline.
• Relevant certifications, such as Certified Information Systems Security Professional (CISSP), Certified Information Security Manager (CISM), Cyber Security Analyst + (CySA+) are highly desirable.

CLEARANCE:

• Secret minimum