cli_filter Plugin API

Overview

This document describes Slurm cli_filter plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own Slurm cli_filter plugins.

The purpose of the cli_filter plugins is to provide programmatic hooks during the execution of the salloc, sbatch, and srun command line interface (CLI) programs. Three hooks are defined:

  • cli_filter_p_setup_defaults — Called before any option processing is done, per job component, allowing a plugin to replace default option values.
  • cli_filter_p_pre_submit — Called after option processing per job component but before any communication is made with the slurm controller. This location is ideal for policy enforcement because the plugin can read all the options supplied by the user (as well as the environment) - thus invalid job requests can be stopped before they ever reach the slurmctld.
  • cli_filter_p_post_submit — Called after the jobid (and, in the case of srun, after the stepid) is generated, and typically before or in parallel with job execution. In combination with data collected in the cli_filter_p_pre_submit() hook, this is an ideal location for logging activity.

    • cli_filter plugins vary from the job_submit plugin as it is entirely executed client-side, whereas job_submit is processed server-side (within the slurm controller). The benefit of the cli_filter is that it has access to all command line options in a simple and consistent interface as well as being safer to run disruptive operations (e.g., quota checks or other long running operations you might want to use for integrating policy decisions), which can be problematic if run from the controller. The disadvantage of the cli_filter is that it must not be relied upon for security purposes as an enterprising user can circumvent it simply by providing an alternate slurm.conf with the CliFilterPlugins option disabled. If you plan to use the cli_filter for managing policies, you should also configure a job_submit plugin to reinforce those policies.

    Slurm cli_filter plugins must conform to the Slurm Plugin API with the following specifications:

    const char plugin_name[]="full text name"

    A free-formatted ASCII text string that identifies the plugin.

    const char plugin_type[]="major/minor"

    The major type must be "cli_filter." The minor type can be any suitable name for the type of job submission package. We include samples in the Slurm distribution for

    • none — An empty plugin with no actions taken, a useful starting template for a new plugin.

    const uint32_t plugin_version
    If specified, identifies the version of Slurm used to build this plugin and any attempt to load the plugin from a different version of Slurm will result in an error. If not specified, then the plugin may be loaded by Slurm commands and daemons from any version, however this may result in difficult to diagnose failures due to changes in the arguments to plugin functions or changes in other Slurm functions used by the plugin.

    Slurm can be configured to use multiple cli_filter plugins if desired, however the lua plugin will only execute one lua script named "cli_filter.lua" located in the default script directory (typically the subdirectory "etc" of the installation directory).

    API Functions

    All of the following functions are required. Functions which are not implemented must be stubbed.

    int init(void)

    Description:
    Called when the plugin is loaded, before any other functions are called. Put global initialization here.

    Returns:
    SLURM_SUCCESS on success, or
    SLURM_ERROR on failure.

    void fini(void)

    Description:
    Called when the plugin is removed. Clear any allocated storage here.

    Returns: None.

    Note: These init and fini functions are not the same as those described in the dlopen (3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the Slurm init(), and the Slurm fini() is called before the system's _fini().

    int cli_filter_p_setup_defaults(slurm_opt_t *options, bool early)

    Description:
    This function is called by the salloc, sbatch, or srun command line interface (CLI) programs shortly before processing any options from the environment, command line, or script (#SBATCH). The hook may be run multiple times per job component, once for an early pass (if implemented by the CLI), and again for the main pass. Note that this call is skipped for any srun command run within an existing job allocation to prevent settings from overriding the set of options that have been populated for the job based on the job environment. The options and early arguments are meant to be passed to slurm_option_set() which will set the option if it is in the appropriate pass. Failures to set an option may be a symptom of trying to set the option on the wrong pass. Given that you should not return SLURM_ERROR simply because of a failure to set an option.

    Arguments:
    options (input) slurm option data structure; meant to be passed to the slurm_option_* API within src/common/slurm_opt.h.
    early (input) boolean indicating if this is the early pass or not; meant to be passed to the slurm_option_* API within src/common/slurm_opt.h.

    Returns:
    SLURM_SUCCESS on success, or
    SLURM_ERROR on failure, will terminate execution of the CLI.

    int cli_filter_p_pre_submit(slurm_opt_t *options, int offset)

    Description:
    This function is called by the CLI after option processing but before any communication with the slurmctld is made. This is after all cli_filter_p_setup_defaults() hooks are executed (for the current job component), environment variables processed, command line options and #SBATCH directives interpreted. cli_filter_p_pre_submit() is called before any parts of the data structure are rewritten, so it is safe to both read and write or unset any options from the plugin that you desire. Note that cli_filter_p_post_submit() cannot safely read (or write) the options, so you should save any state for logging in cli_filter_p_post_submit() during cli_filter_p_pre_submit(). This function is called once per job component.

    Arguments:
    options (input/output) the job allocation request specifications.
    offset (input) integer value for the current hetjob offset; should be used as a key when storing data for communication between cli_filter_p_pre_submit() and cli_filter_p_post_submit().

    Returns:
    SLURM_SUCCESS on success, or
    SLURM_ERROR on failure, will terminate execution of the CLI.

    void cli_filter_p_post_submit(int offset, uint32_t jobid, uint32_t stepid)

    Description:
    This function is called by the CLI after a jobid (and, if srun, a stepid) has been assigned by the controller. It is no longer safe to read or write to the options data structure, so it has been removed from this function. You should save any state you need in cli_filter_p_pre_submit() using het_job_offset as a key, since the function is called separately for every job component, and access it here.

    Arguments:
    offset (input) integer value for the current hetjob offset; should be used as a key when storing data for communication between cli_filter_p_pre_submit() and cli_filter_p_post_submit().
    jobid (input) job id of the job
    stepid (input) step id of the job if appropriate, NO_VAL otherwise

    LUA Interface

    Setting CliFilterPlugins=cli_filter/lua in slurm.conf will allow you to implement the API functions mentioned using Lua language. The file must be named "cli_filter.lua" and, similar to the job_submit plugin, it must be located in the default configuration directory (typically the subdirectory "etc" of the installation). An example is provided within the source code here.

    NOTE: Although available options are defined in the struct slurm_opt_t within src/common/slurm_opt.h, some options might be renamed. The provided example shows a way of displaying the configured options by using slurm.json_cli_options(options).

    User Defaults

    Setting CliFilterPlugins=cli_filter/user_defaults in slurm.conf will allow users to define their own defaults for jobs submitted from the machine(s) with the configured file. The plugin looks for the definition file in $HOME/.slurm/defaults. It will read each line as a component=value pair, where component is any of the job submission options available to salloc, sbatch, or srun and value is a default value defined by the user. The following example would configure each job to have a default name, time limit, amount of memory, and error and output files: 

    job-name=default_name
time=10:00
mem=256
error = slurm-%j.errfile
output = slurm-%j.logfile

    You can also specify different default settings for jobs based on the command being used to submit the job and/or the cluster being submitted to. The syntax for this would be:
    <command>:<cluster>:<component>

    <command> could be one of:

    • salloc: Jobs submitted with the salloc command.
    • sbatch: Jobs submitted with the sbatch command.
    • srun: Jobs submitted with the srun command.
    • *: Jobs submitted with any submission command.

    <cluster> could be any defined cluster on your system, or * to have it match any cluster.

    <component> is any of the job submission options available to salloc, sbatch, or srun.

    The following example would assign different default partitions based on the command used to submit the job. It would also assign different partitions for jobs submitted with salloc, depending on the cluster being used: 

    salloc:cluster1:partition = interactive
salloc:cluster2:partition = member
sbatch:*:partition = high
srun:*:partition = short

    Last modified 19 February 2024