build an OCaml module for a Lua program

[epatrizio]

Hi @pdonadeo and thanks for this work!

I am currently studying the Lua language in an academic context:

• OCaml Lua Interpreter - experimental, wip: only for a better understanding of lua syntax, typing, execution model, etc.
• Lua skeleton - wip: some minimal concrete examples in different contexts

My current exercise consists of building an OCaml module (a simple factorial function) and calling it in a Lua program :

• https://github.com/epatrizio/lua-skeleton/blob/main/src/factorial.ml
• https://github.com/pdonadeo/ocaml-lua/blob/master/examples/lua_book/dir.ml (source of inspiration)

local factorial_ml = require("factorial_ml")
require call launches this error :

error loading module 'factorial_ml' from file './factorial_ml.so':
./factorial_ml.so: undefined symbol: caml_termination_hook

It seems that I have an error in my compilation process during ocaml runtime linking :
https://github.com/epatrizio/lua-skeleton/blob/main/Makefile : fact_ocaml_build rule

For other informations and details :
epatrizio/lua-skeleton#1

Could you help me? Is it really a compilation problem? Or something else?

Thanks a lot in advance

[pdonadeo]

Hi @epatrizio, happy to help if I can.

I don’t fully understand what you’re trying to do with the factorial module. What’s the program’s “main”? To be more specific: is the main in OCaml and you want to call functions written in Lua, or is the main in Lua and you want to call functions written in OCaml as libraries available in the Lua state?

I see several OCaml and C files, and I haven’t figured out where the program execution starts or where you want it to start.

In my example you mentioned, dir.ml, taken directly from Programming in Lua, 2nd Edition, chapter 26, the “main” is in Lua and functions written in C are called.

I also don’t see any typical dune files: dune and dune-project, so I don’t understand how to build your experiment.

[epatrizio]

Thanks for the quick response.

My goal is to call OCaml functions in a Lua program.

Don't try to understand Lua skeleton, this repo is WIP (draft state, no documentation, ...).
To summarize, I have a working example of a C module that is called in a Lua program :

• https://github.com/epatrizio/lua-skeleton/blob/main/src/factorial.c : the C module
• https://github.com/epatrizio/lua-skeleton/blob/main/spec/factorial_spec.lua : C module call in the test suite (load: line 4 local factorial_c = require("factorial_c"))

Now, I try to make exactly the same thing in OCaml

• https://github.com/epatrizio/lua-skeleton/blob/main/src/factorial.ml : the OCaml module
• https://github.com/epatrizio/lua-skeleton/blob/main/spec/factorial_spec.lua : OCaml module call in the test suite (load: line 6 in comment -- local factorial_ml = require("factorial_ml"))

My problem is (it seems to me) the OCaml compilation to produce the .so shared library.

The C module (most common approach) everything is ok with luarocks automation :
https://github.com/epatrizio/lua-skeleton/blob/main/lua-skeleton-0.1-1.rockspec : build entry, line 27

gcc -O2 -fPIC -I/usr/local/include -c src/factorial.c -o src/factorial.o
gcc  -shared -o /tmp/luarocks_build-lua-skeleton-0.1-1-7617795/factorial_c.so src/factorial.o

Now, I try to compile OCaml module by hand (not in a dune project) inspired by C example :
https://github.com/epatrizio/lua-skeleton/blob/main/Makefile

ocamlfind ocamlopt -output-obj -o factorial_ml.o src/factorial.ml \
-linkpkg -package ocaml-lua \
-thread; \
gcc -L`ocamlopt -where` -shared -o factorial_ml.so factorial_ml.o \
`ocamlopt -where`/../ocaml-lua/liblua_stubs.a -lasmrun_pic -lthreadsnat -lunix -lm -ldl

but loading module launches :

error loading module 'factorial_ml' from file './factorial_ml.so':
./factorial_ml.so: undefined symbol: caml_termination_hook

Sorry for all the details, I hope it's clearer now. Thanks.

[pdonadeo]

Ok, now I understand the point. Unfortunately, I have never tried an example using luaL_register, so I need some time to put together a working example in OCaml — probably over the coming weekend.

[epatrizio]

probably over the coming weekend

no problem, no urgency.

[pdonadeo]

Hi @epatrizio, unfortunately the short answer is no: with my library it’s not possible to write an OCaml library, compile a .so, and load it into the Lua interpreter with require.

Here’s the longer explanation. The library was designed to embed a Lua interpreter inside an OCaml application, so the application can expose a configuration language to its users — similar to what Neovim or Wireshark do.

The crucial part of the binding I wrote — and the main difficulty I had to solve — is making the two garbage collectors coexist: OCaml’s GC and Lua’s GC. From OCaml you can create a Lua state, pass any value to Lua, create Lua objects there and pass them back to OCaml, while being sure that neither GC will free something the other side still considers live.

To make this work, when I create a Lua state using LuaL.newstate, I use a custom allocator, as you can see in the C implementation of newstate here:

ocaml-lua/src/lua_aux_lib_stubs.c

Lines 267 to 313 in f44ad50

	CAMLprim
	value luaL_newstate__stub (value max_memory_size, value unit)
	{
	CAMLparam2(max_memory_size, unit);
	CAMLlocal2(v_L, v_L_mirror);
	debug(3, "luaL_newstate__stub: BEGIN\n");
	value *default_panic_v = caml_named_value("default_panic");
	/* alloc space for the register entry */
	ocaml_data *data = (ocaml_data*)caml_stat_alloc(sizeof(ocaml_data));
	/* protect the panic_callback portion and assign with the default value */
	caml_register_global_root(&(data->panic_callback));
	data->panic_callback = *default_panic_v;
	/* create a fresh new Lua state */
	int max_memory = Int_val(max_memory_size);
	/* init the allocator data */
	data->ad.max_memory = max_memory;
	data->ad.used_memory = 0;
	lua_State *L = lua_newstate(custom_alloc, (void*)&(data->ad));
	debug(5, "luaL_newstate__stub: lua_newstate returned %p\n", (void*)L);
	debug(6, " luaL_newstate__stub: calling lua_atpanic...");
	lua_atpanic(L, &default_panic);
	debug(6, " done!\n");
	/* wrap the lua_State* in a custom object */
	v_L = caml_alloc_custom(&lua_State_ops, sizeof(lua_State *), 1, 10);
	lua_State_val(v_L) = L;
	/* another value wrapping L for internal purposes */
	v_L_mirror = caml_alloc_custom(&default_lua_State_ops, sizeof(lua_State *), 1, 10);
	lua_State_val(v_L_mirror) = L;
	caml_register_global_root(&(data->state_value));
	data->state_value = v_L_mirror;
	/* create a new Lua table for binding informations */
	create_private_data(L, data);
	debug(4, "luaL_newstate__stub: RETURN %p\n", (void*)v_L);
	/* return the lua_State value */
	CAMLreturn(v_L);
	}

But when you start the Lua REPL and type require "mylib", the Lua state is created by Lua itself using its default allocator. In that scenario it becomes impossible — or at least, I have no idea how to make the two garbage collectors interact safely.

If you have any idea on how to tackle this, I’d be happy to try implementing an “adapter” function that maps a raw C pointer to a Lua state (allocated by Lua) to one managed by the binding.

[epatrizio]

Hi @pdonadeo, Thank you for the detailed response.
I'm interested in thinking about it, it's a good exercise (very informative ;-)
I have to be away for a few days, but I'll get back to it after that.

[epatrizio]

Hi @pdonadeo!
I'm starting to understand how difficult it is to make the two GC coexist. For now, I'm also stuck.
Nevertheless, for now, I'm not convinced that my OCaml compilation problem is related ?

error loading module 'factorial_ml' from file './factorial_ml.so':
./factorial_ml.so: undefined symbol: caml_termination_hook

These topics are complicated for me, so I'll get back to them after the end-of-year break ;)