How to use job_submit_lua plugin with Slurm ?

Lua is scripting language implemented as a C library, which makes it perfect choice for small plugins to bigger C applications. It’s both efficient (since it’s really a C library, lua file can be read once and functions are normally executed multiple times) and easy to modify.

In case of queuing system like Slurm[1] there is always a need for customization that will fulfill organization specific requirements – like some preliminary checks done on job submission. In this case Slurm framework offers so called job_submit plugins, which are normally a shared libraries (.so) implementing two required functions job_submit and job_modify and two optional init and fini. While compiling .c into shared library is not a big deal there are situations where one would rather use script version, for those Slurm provides job_submit_lua plugin that simply calls lua script for real work. In this post I’ll cover the issues I met configuring it.

How to use job_submit_lua plugin in Slurm?

Compile Slurm with job_submit_lua.so

When you run ./configure it has to discover lua libraries to link against them. In my case (Centos 6), when I just installed lua-devel (yum -y install lua-devel) and executed./configure I noticed

configure: WARNING: unable to locate lua package

and in config.log

[...]Package lua was not found in the pkg-config search path.                                                                                                                
Perhaps you should add the directory containing `lua.pc'                                                                                                                
to the PKG_CONFIG_PATH environment variable                                                                                                                             
No package 'lua' found                                                                                                                                                  
configure:24443: $? = 1                                                                                                                                                 
configure:24457: result: no                                                                                                                                             
No package 'lua' found                                                                                                                                                  
configure:24533: WARNING: unable to locate lua package
[...]

Simple check with manual execution of pkg-config, confirmed that it doesn’t exist

[root@hpc-slurmtest slurm-17.11.7]# pkg-config --exists --print-errors "lua-5.1"                                                                                        
Package lua-5.1 was not found in the pkg-config search path.                                                                                                            
Perhaps you should add the directory containing `lua-5.1.pc'                                                                                                            
to the PKG_CONFIG_PATH environment variable                                                                                                                             
No package 'lua-5.1' found

OK.. maybe CentOS 6 is too old and it has older version of lua, check:

[root@hpc-slurmtest slurm-17.11.7]# yum info lua-devel     | grep '^Version'                                                                                                             
Version     : 5.1.4

So maybe it doesn’t provide a .pc (package config file)

[root@hpc-slurmtest slurm-17.11.7]# rpm -ql lua-devel | grep ‘.pc’                           
/usr/lib64/pkgconfig/lua.pc

File is there, but it’s name is not lua-5.1 or lua5.1 it’s just lua, let’s check if symlink creation will help the .m4 macro to find it:

ln -s  /usr/lib64/pkgconfig/lua.pc /usr/lib64/pkgconfig/lua-5.1.pc
./configure
[...]
checking for lua... yes
checking for whether we can link to liblua... yes lua
[...]

I tried to check how the macro is implemented, you can find it in auxdir/x_ac_lua.m4. I rewritten it to more standard use of PKG_CHECK_MODULE and asked SchedMD for opinion[2]. Additional research revealed that lua development team doesn’t provide pkg-config files[3]. They are created by distribution packagers, so they may differ between operating systems and the dirty implementation in Slurm autotools macro may be just fixing incompatibility issues.

Remember that if you play with macros used by autotools you have to recreated configure. Slurm follows quite standard way of doing it by make configure, but it looks like Makefile dependencies are not implemented correctly. I had to remove configure manually, because make didn't notice changes in x_ac_lua.

Where should I put my .lua script?

I got it compiled so could switch to implementation of my .lua script, but where to create it? Checking slurm.conf manual in section about JobSubmitPlugin you’ll find explanation that was quite enigmatic, at least for me when I read it:

JobSubmitPlugin
[...]
For examples of use, see the Slurm code in "src/plugins/job_submit" and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs. Slurm can be configured to use multiple job_submit plugins if desired, however the lua plugin will only execute one lua script named "job_submit.lua" located in the default script directory (typically the subdirectory "etc" of the installation directory). No job submission plugins are used by default.

When I read this for the first time I was unable to understand where should I put my job_submit.lua, but quick check on src/plugin/job_submit/lua/job_submit_lua.c shown that location comes from DEFAULT_SCRIPT_DIR, which is

[root@hpc-slurmtest slurm-17.11.7]#  grep -ri  DEFAULT_SCRIPT_DIR ./src/plugins/job_submit/lua/Makefile
./src/plugins/job_submit/lua/Makefile:AM_CPPFLAGS = -DDEFAULT_SCRIPT_DIR=\"$(sysconfdir)\" \

Simply ${prefix}/etc in other words next to slurm.conf and I agree documentation stated it clearly 🙂

Develop the plugin

My goal was to develop a plugin that will prevent submission of jobs without explicit account specification (we found out that users don’t adjust account specification simply relying on the default, which caused some internal billing issues). I simply copied the example, changed the job_submit function to very simple:

function slurm_job_submit(job_desc, part_list, submit_uid)

        if job_desc.account == nil then
                slurm.log_user("You have to specify account. Usage of default accounts is forbidden.")
                return slurm.ESLURM_INVALID_ACCOUNT
        end
end

Added JobSubmitPlugins=lua line to slurm.conf, restarted slurmctld and attempted to test it by srun --pty bash -l. Surprisingly error message, one from slurm.log_user(), was displayed but the job started, which means that my return statement didn’t really return an error. I just used ESLURM_INVALID_ACCOUT since it’s one of the error values defined in slurm/slurm_error.h and was the most suitable for me. However, how is it passed to lua ?

Dig into Slurm code one more time you'll see it’s passed to the lua library in job_submit plugin init function and not all error codes are there. Now the sentence I like: "It’s open source so just add what you need". One more time create a patch and send it to main development team to check if it's fine and to be able to use my script after Slurm upgrade. It was accepted in a few hours[4] and job_submit script worked as expected:

[root@hpc-slurmtest slurm]# srun --pty bash -l
srun: error: You have to specify account. Usage of default accounts is forbidden.
srun: error: Unable to allocate resources: Invalid account or account/partition combination specified

[1] https://slurm.schedmd.com/
[2] https://bugs.schedmd.com/show_bug.cgi?id=5263
[3] http://lua-users.org/lists/lua-l/2010-05/msg00641.html
[4] https://github.com/SchedMD/slurm/commit/2aa74b52106ff41a39e849839774e6e499807a1e