This prometheus exporter hosts different collectors, identified by a pair of labels: monitoring-id and hpc label. Each collector connects via ssh to the frontend of a given HPC infrastructure and queries the scheduler in order to collect and expose the metrics of all the jobs the collector in question must monitor (through job_ids passed to it). It also gathers and exposes data on the HPC partitions/queues. Each collector labels all their metrics with the labels monitoring_idand hpc.

The ssh credentials for the hpc are retrieved from Vault, using the user's JWT to authenticate.

The exporter offers an API with the following endpoints:

• /collector [POST]: Creates a collector. Authorization header must contain a valid JWT. Payload is a JSON object containing the configuration for the collector:

  {
    "host": <HPC frontend URL>,
    "scheduler": <"slurm" or "pbs">,
    "auth_method": <"keypair" or "password">,
    "sacct_history": <Days back to query for metrics, only for slurm>,
    "scrape_interval": <Maximum query interval, in seconds>,
    "deployment_label": <Human-readable label to identify a deployment, will be included in all the metrics as a label>,
    "monitoring_id": <Unique identifier for the deployment, will be included in all the metrics as a label>,
    "hpc_label": <Human-readable to identify the HPC this collector is connected to, , will be included in all the metrics as a label>,
    "only_jobs": <Boolean. If True, infrastructure metrics will not be collected>,
  }
  

• /collector [DELETE]: Deletes a collector. Authorization header must also contain a valid JWT. Only the user that created the collector can delete it. Payload is a JSON object containing information to identify the collector:

  {
    "host": <HPC frontend URL>,
    "monitoring_id": <Unique identifier for the deployment>,
  }
  

• /job [POST]: Adds a jobid to the list of jobs a collector monitors. Payload is a JSON object containing information to identify the collector, as well as the jobid itself:

  {
    "host": <HPC frontend URL>,
    "monitoring_id": <Unique identifier for the deployment>,
    "job_id": <Job ID to add tot he collector>
  }
  

Labels of all the job metrics: {job_id, job_name, job_user, job_queue} :

• pbs_job_state: Job state numeric code. See State codes below for details.
• pbs_job_state: Exit status numeric code
• pbs_job_priority: Job current priority
• pbs_job_walltime_used: Walltime consumed in seconds
• pbs_job_walltime_max: Maximum wall time in seconds
• pbs_job_walltime_remaining: Remaining wall time in seconds
• pbs_job_cpu_time: Consumed CPU time in seconds
• pbs_job_cpu_n: Number of threads requested by this job
• pbs_job_memory_physical: Physical memory consumed in bytes
• pbs_job_memory_virtual: Virtual memory consumed in bytes
• pbs_job_time_queued: Time spent in queue by the job in seconds
• pbs_job_exit_status: Job exit status. -50 if not completed. Check here for the meaning of each code

Labels of all the queue metrics: {queue_name, queue_type} :

• pbs_queue_enabled: 1 if the queue is enabled, 0 if it is disabled
• pbs_queue_started: 1 if the queue is started, 0 if it is stopped
• pbs_queue_jobs_max: Maximum number of jobs that can be run in the queue (-1 if inf)
• pbs_queue_jobs_queued: Number of jobs with queued status in the queue
• pbs_queue_jobs_running: Number of jobs with running status in the queue
• pbs_queue_jobs_held: Number of jobs with held status in the queue
• pbs_queue_jobs_waiting: Number of jobs with waiting status in the queue
• pbs_queue_jobs_transit: Number of jobs with transit status in the queue
• pbs_queue_jobs_exiting: Number of jobs with exiting status in the queue
• pbs_queue_jobs_complete: Number of jobs with complete status in the queue

Labels of all the job metrics: {job_id, job_name, job_user, job_partition} :

• slurm_job_state: Job state numeric code. State codes below for details.
• slurm_job_walltime_used: Walltime consumed in seconds
• slurm_job_cpu_n: Number of CPUs assigned to this job
• slurm_job_memory_physical_max: Maximum physical memory in bytes that can be allocated to this job
• slurm_job_memory_virtual_max: Maximum virtual memory in bytes that can be allocated to this job
• slurm_job_queued: Time spent in queue by the job in seconds

Labels of all the partition metrics: {queue_name} :

• slurm_partition_availability: Availability status code of the partition. See State codes below for details.
• slurm_partition_cores_total: Number of cores in this partition
• slurm_partition_cores_alloc: Number of cores allocated to a job in this partition
• slurm_partition_cores_idle: Number of idle cores in this partition

1. Download the code
2. Enter the folder and build it with go

go build

3. Run the exporter

hpc_exporter -listen-address <PORT> -log-level <LOGLEVEL> -introspection-endpoint <OIDC_INTROSPECTION_ENDPOINT> -introspection-client <OIDC_INTROSPECTION_CLIENT> -introspection_secret <OIDC_INTROSPECTION_SECRET> -vault-address <VAULT_ADDRESS>

• <PORT>: Port the metrics will be exposed on for prometheus. :9110 as default.
• <LOGLEVEL>: Logging level. error as default, warn, info and debug also supported
• <OIDC_INTROSPECTION_ENDPOINT>: Endpoint of the service to verify JWTs. Default is env variable OIDC_INTROSPECTION_ENDPOINT
• <OIDC_INTROSPECTION_CLIENT>: Client of the service to verify JWTs. Default is env variable OIDC_INTROSPECTION_CLIENT
• <OIDC_INTROSPECTION_SECRET>: Client secret of the service to verify JWTs. Default is env variable OIDC_INTROSPECTION_SECRET
• <VAULT_ADDRESS>: Address of the Vault instance that holds the ssh credentials. Default is env variable VAULT_ADDRESS

Check the README.md file in docker folder for instructions about Docker deployment of HPC exporter

There are 2 authentication methods supported. Both require that the credentials are saved in the Vault instance the exporter is configured to access. The credentials must be stored in a secret in /hpc/<username>/<hpc-address>

• password: Password authentication. Expects the password to be in the secret with the key password.
• keypair: Public-private key authentication. Expects the private key to be stored in the vault secret with the key pkey in plain text. The content of the value must include header and footer such as -----BEGIN RSA PRIVATE KEY-----\n and \n-----BEGIN RSA PRIVATE KEY-----. There must be no line breaks (/n) inside the key itself.

The state of each job and partition is reported as a numerical value. The correspondence between the state and the code is as follows. For more information about what each state means, consult Slurm and PBS official documentation.