Senior HPC Systems Programmer - 139214

DEPARTMENT OVERVIEW:

The Mission of the San Diego Supercomputer Center is to translate innovation into practice. SDSC adopts and partners on innovations in industry and academia in the areas of software, hardware, computational and data sciences, and related areas, and translates them into cyberinfrastructure that solves practical problems across any and all scientific domains and societal endeavors. Cyberinfrastructure refers to an accessible, integrated network of high-performance computing, data, and networking resources and expertise, focused on accelerating scientific inquiry and discovery. With more than 250 employees and $30-50M of revenue a year, SDSC is a global leader in the design, development, and operations of cyberinfrastructure.

SDSC supports hundreds of multidisciplinary programs spanning a wide variety of domains, from earth sciences and biology to astrophysics, bioinformatics, and health IT. SDSC presently operates multiple large HPC systems ranging from a 120k x86 CPU core general purpose system to a system explicitly designed for Artificial Intelligence and Machine Learning, and a nationally distributed system open for all of academia to integrate with. SDSC offers research data services across the entire vertical stack from universally scalable storage to consulting services on FAIR, Big Data, and AI. SDSC offers a rich set of cloud services both on-premise, in the commercial cloud, and as hybrid services across both.

SDSC has three geographic scopes, a national scope supporting cyberinfrastructure for the entire US research and education community, a California scope with a special focus on convergence research that addresses the three dominant threats to CA: Drought, Fire, Earthquakes, and a campus scope focusing on advancing the global impact of SDSC by advancing the research objectives of the UC San Diego faculty, researchers, and students.

SDSC impacts researchers at scales from 1,000's to Millions. SDSC annually trains thousands of researchers in cyberinfrastructure tools and software, and supports thousands of individual researchers via Unix accounts on its large HPC systems. SDSC was a leader developing the Science Gateway concept, and continues to be a global leader in its evolution. SDSC operates multiple major such gateways with user communities ranging from the tens of thousands to the millions. SDSC's educational programs includes online courses that have been attended by more than a million students.

SDSC is committed to democratizing access to cyberinfrastructure across all of its geographic scopes. SDSC strives towards a culture that supports our employees to be their best, achieve their goals, and enjoy their lives, both professionally and personally.

The Data Enabled Scientific Computing (DESC) division within SDSC designs and jointly proposes with other SDSC researchers, supercomputing systems in response to tens of millions of dollars call-for-proposals from the National Science Foundation (NSF), various government organizations and UC entities; it responds to calls for proposals for cyberinfrastructure (CI) related research, solutions and support. DESC manages, operates and troubleshoots issues with advanced, leading edge, complex, multi-petaflop and multi-petabyte data intensive supercomputer systems, file systems (Lustre, Ceph etc.), interconnects (such as InfiniBand, NVLink, ethernet etc.) and CI projects housed at SDSC. Research leaders within DESC submit high performance computing (HPC), high throughput computing (HTC), AI, CI, data science, computational science, science gateways and scientific software research proposals and acquire funding from NSF, National Institutes of Health (NIH), Department of Energy (DOE), Department of Defense (DOD) and industry. DESC carries out supercomputing, CI, data science, computational science and scientific software research and development projects. This division provides consulting and user support to researchers and users from academia as well as collaborates with them and industrial users. DESC provides advanced computational science, CI and scientific software support for the national and UC user communities as a part of projects/machines such as the Expanse machine (a five-year ~$25-million project supported by the NSF and enables tens of thousands of users to use HPC, HTC and GPUs), Voyager machine (a five-year, ~$12-million project supported by the NSF and enables researchers to experiment with and use AI-focused hardware for scientific applications), PNRP project ( a five-year , ~$12-million project supported by the NSF and enables distributed computing with resources of GPUs, FPGAs and CPUs), the Triton Shared Compute Cluster (TSCC - which is a condo cluster primarily for UCSD researchers), and the CloudBank project ( a five-year, ~$6-million project supported by the NSF to Simplify Cloud Access for Computer Science Research and Education). Various other funded CI research and development, and domain science (e.g. biochemistry, bioinformatics, cosmology etc.) projects are directed by DESC researchers. DESC staff and researchers are involved with the NSF funded Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program. DESC is involved in various HPC/HTC/AI training, workshop, outreach, workforce development and K-12 student programs and associated NSF funded projects. This division stays current with HPC, HTC, accelerators, CI, computational science and scientific software research and technology trends and engages with supercomputer vendors ?(e.g. Dell, Supermicro, Intel, NVIDIA, AMD, IBM, Hewlett Packard Enterprise, Data Direct Network, Aeon Computing, Arista etc.) to remain current with future technologies utilized in supercomputer designs.

The Senior HPC Systems Programmer will apply advanced software concepts and objectives to medium to large projects of broad scope and complexity, regularly resolve highly complex issues where analysis of situations or data requires an in-depth evaluation of variable factors as well as select tools, methods, techniques and evaluation criteria to obtain results.

As the scale and complexity of HPC systems grow, the ability to automate systems management and to understand, analyze, and troubleshoot multiple hardware and software layers becomes critical. As part of SDSC's High Performance Systems group, the Senior HPC Systems Programmer performs software design and is a lead developer in the areas of: analyzing network and file system performance; automating system maintenance; monitoring systems and network; and developing innovative cyberinfrastructure for campus, state, and national scale HPC platforms and Cloud infrastructure orchestration. The position also participates in the analysis, specification, design, implementation, and monitoring of high performance and cloud computing systems to enable scientific research computing.

The responsibilities include management of the HPC cluster and cloud software development environment, defining coding standards, and code review. As part of the design process, creating software architectures with an eye for scaling to the thousands of computers in production. Leads and automates the testing of complex software platforms operating across several networks and after validating the changes, coordinate the software update of production HPC clusters. Leads development, deployment and management of Cloud and container infrastructure projects, extensions to existing software tools and creates and applies policies for resource use. The developed and maintained software manages and monitors system internals, networks, storage, and operating systems, along with emerging technologies. Supporting this requires in-depth knowledge and experience with all the facets of HPC and cloud systems, such as resource managers, schedulers, orchestrators, and parallel file systems (e.g., Lustre). The development tasks specifically include managing Kubernetes clusters like the PNRP Nautilus and the Voyager AI supercomputer.

The position involves working closely with the user services teams to continually improve the process for building and deploying HPC applications. As part of the HPC Systems Group, refines and updates the systems management and monitoring software to provide a constantly improving level of availability of our clusters and file systems. Determines features needed for new and existing software, chooses best methods to implement those features, sets the task priority and communicates those tasks to other members of the team. Prioritizes multiple simultaneous software development project based on need and communicates resource requirements to management and project leaders. Works with User Services to continually improve the process for building and deploying applications. This includes working with the PNRP systems and user support teams to evaluate new feature requirements, designing solutions and implementing them.

The incumbent is responsible for incorporating application testing into automated build processes. This includes development, deployment, and integration of system tests into the operation of clusters. Must ensure the appropriate level of security of the development, staging, and production environments to facilitate rapid early development while maintaining secure build environments for production software. Defines and implements the process for staging and testing changes to production based on the invasiveness of the change.

They will participate in collaborative, team-based efforts, including national initiatives such as ACCESS and related working groups and prioritize a complex set of development and deployment tasks based on input from User Services and management. The incumbent understand the requirements of highly complex distributed services that interact across multiple internal and external projects at the SDSC, campus, and National level and will provide guidance and mentorship to junior staff, supporting their development in HPC systems operations and large-scale data management. They will also work closely with other SDSC and UCSD groups to integrate HPC compute and storage systems into broader campus and national research activities. The Senior HPC Systems Programmer will create software specifications with focuses on availability, reliability, perfomance, scaling, and maintainability by others within DESC, work with teams from national projects in a distributed development environment and contribute to the development and implementation of security practices and operational procedures. Additionally, they will partner with User Services and operations teams to support training, documentation, and coordination of system maintenance activities as well as contribute to technical presentations, documentation, and knowledge sharing within the organization and at professional meetings.

For more information, please visit: https://www.sdsc.edu/

• Bachelor's degree in related area and / or equivalent experience / training.

• Advanced knowledge of secure software development utilizing diverse programming languages including but not limited to C/C++, Java, Javascript, Python and Go.

• Strong demonstrated experience with Ethernet and IP networking, including VLANs, subnets, and routing.

• Advanced knowledge of Linux, including services, networking, and file systems.

• Advanced experience managing Kubernetes clusters, with specific focus on geographically distributed designs.

• Demonstrated experience setting up AI infrastructure including incorporation of innovative accelerator hardware, storage, and inference services leveraging Kubernetes and on-prem HPC resources

• Job offer is contingent upon satisfactory clearance based on Background Check results.

• Occasional evenings and weekends may be required. On-call rotation may be required.

If employed by the University of California, you will be required to comply with our Policy on Vaccination Programs, which may be amended or revised from time to time. Federal, state, or local public health directives may impose additional requirements.

To foster the best possible working and learning environment, UC San Diego strives to cultivate a rich and diverse environment, inclusive and supportive of all students, faculty, staff and visitors. For more information, please visit UC San Diego Principles of Community.

The University of California is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, protected veteran status, or other protected status under state or federal law.

For the University of California's Anti-Discrimination Policy, please visit: https://policy.ucop.edu/doc/1001004/Anti-Discrimination

UC San Diego is a smoke and tobacco free environment. Please visit smokefree.ucsd.edu for more information.

Misconduct Disclosure Requirement: As a condition of employment, the final candidate who accepts an offer of employment will be required to disclose if they have been subject to any final administrative or judicial decisions within the last seven years determining that they committed any misconduct; or have filed an appeal of a finding of substantiated misconduct with a previous employer.

a. "Misconduct" means any violation of the policies governing employee conduct at the applicant's previous place of employment, including, but not limited to, violations of policies prohibiting sexual harassment, sexual assault, or other forms of harassment, or discrimination, as defined by the employer. For reference, below are UC's policies addressing some forms of misconduct:

• UC Sexual Violence and Sexual Harassment Policy
• UC Anti-Discrimination Policy
• Abusive Conduct in the Workplace