Performance Test Engineer- LGY

Performance Test Engineer/Developer- LGY

Location: Remote

Description: Seeking a Performance Test Engineer/Developer to support the Department of Veterans Affairs (VA) Loan Guaranty Service (LGY). The Performance Test Engineer/Developer will be responsible for building the observability & monitoring artifacts, telemetry and logging that supports serverless system performance, diagnosing performance bottlenecks, and supporting performance optimization efforts across the LGY platform. The Performance Developer will not just write Lambda code, they will own the engineering pattern for how LGY Lambda services are built, instrumented with AWS ADOT, deployed through CI/CD, tested, and monitored in Dynatrace & Splunk. This role works closely with development first and then assists operations teams to ensure system performance scalability, and reliability across Salesforce and AWS-based integration services. The ideal candidate must have development experience working with API-driven systems, cloud infrastructure, and performance monitoring tools to develop & analyze telemetry, logs, and system metrics to identify and resolve performance issues.

Job Responsibilities:

• Design, build, & maintain AWS Lambda services in Node.js & Python based serverless portfolio.
• Lead AWS ADOT-based instrumentation for Lambda workloads, including OpenTelemetry layer adoption, configuration, OTLP settings, & runtime compatibility validation.
• Use GitHub actions develop pipelines to standardize CDK Lambda constructs for Dynatrace & AWS ADOT libraries, reusable layers, environment variables, active tracing, Lambda Insights, and centralized log configuration.
• Validate Dynatrace instrumentation compatibility for Nodejs Function & implement wrapper or alternate instrumentation strategies where required.
• Develop and maintain correlation ID and trace propagation across Lambda, Step Functions, and downstream service calls so transactions are observable end to end.
• Enforce observability practices for AWS cloud workloads, including PII-safe structured logging
• Build & maintain GitHub Actions pipelines for linting, unit tests, coverage, deploy, and post-deployment telemetry validations.
• Create automated tests that verify instrumentation is working, including trace/log/metric arrival in Dynatrace & Splunk, cold-start visibility, and alert validation in each environment.
• Develop python-based solutions to capture Lambda errors duration, errors, throttles, retries, cold starts, workflow failures, and downstream latency.
• Tune Lambda memory, timeout, concurrency, packaging, and network configuration to balance AWS ADOT overhead, Dynatrace visibility, reliability, and cost.
• Modernize older Lambda implementations and pipelines to a common AWS ADOT plus Dynatrace standard across multiple env.
• Investigate performance incidents and collaborate with development teams to diagnose root causes.
• Review system logs, API metrics, and AWS service telemetry to identify infrastructure or application performance constraints.
• Provide performance optimization recommendations to development teams when performance bottlenecks are identified.

Required Skills and Experience:

• Development background experience with Nodejs and Python on serverless architecture. THIS IS A MUST
• Must have develpment background experience implementing & supporting end-to-end tracing, metrics development and structured logging across API Gateway, Step Functions, SQS, S3, RDS/Postgres, Salesforce integrations, Oracle/SOAP services, document APIs.
• Experience with Dynatrace and Splunk
• Experience analyzing application performance metrics and system telemetry.
• Must be able to decipher code for troubleshooting performance issues in API-driven or distributed systems.
• Familiarity with cloud-based environments such as AWS.
• Strong analytical and problem-solving skills.
• Experience working with monitoring tools and log analysis platforms
• Must be a US Citizen or Permanent Resident
• Must be able to obtain and maintain a Public Trust Clearance

Desired Skills and Experience:

• Experience supporting VA or federal government IT systems.
• Experience with AWS services such as Lambda, API Gateway, and cloud monitoring tools.
• Familiarity with Salesforce-based applications and enterprise API integrations.
• Experience supporting systems that process sensitive or regulated data.

Education and Experience:

• Bachelor's degree in Computer Science, Information Technology, or related field or equivalent experience.
• Years of experience in performance engineering, application performance monitoring, or system performance analysis.

#CJ

#dice

About HigherEchelon, Inc.

HigherEchelon, Inc. (HE) is a service-disabled veteran-owned small business (SDVOSB) with offerings in Engineering, Gaming, Human Capital, Enterprise Technology and Cyber Solutions. HigherEchelon aims to be the premier trusted partner in organizational excellence and achieves this through sustained investment in talent and the employee experience. HigherEchelon puts the employee first to better serve customer needs and sustain excellence.

HigherEchelon offers competitive full-time benefits including paid vacation and holidays, 401(k) matching, full health/dental/vision coverage, plus much more. For on-site and remote employees, flexible work schedules are offered when authorized.

By joining our team, you are choosing to embark on a journey towards excellence as a valued team member and trusted partner. We appreciate your inquiry and look forward to discussing the opportunity further.

EOE/Minorities/Females/Veterans/Disabled:

HigherEchelon, Inc. is an Equal Employment Opportunity employer and provides reasonable accommodation for qualified individuals with disabilities and disabled veterans in its job application procedures.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, genetic information, citizenship, ancestry, marital status, protected veteran status, disability status or any other status protected by federal, state, or local law. HigherEchelon, Inc. participates in E-Verify.