# Interfacing C with Caml Light

This chapter describes how user-defined primitives, written in C, can be added to the Caml Light runtime system and called from Caml Light code.

## Overview and compilation information

### Declaring primitives

User primitives are declared in a module interface (a `.mli` file), in the same way as a regular ML value, except that the declaration is followed by the `=` sign, the function arity (number of arguments), and the name of the corresponding C function. For instance, here is how the `input` primitive is declared in the interface for the standard library module `io`:

            value input : in_channel -> string -> int -> int -> int
                        = 4 "input"

Primitives with several arguments are always curried. The C function does not necessarily have the same name as the ML function.

Values thus declared primitive in a module interface must not be implemented in the module implementation (the `.ml` file). They can be used inside the module implementation.

### Implementing primitives

User primitives with arity n <= 5 are implemented by C functions that take n arguments of type `value`, and return a result of type `value`. The type `value` is the type of the representations for Caml Light values. It encodes objects of several base types (integers, floating-point numbers, strings, ...), as well as Caml Light data structures. The type `value` and the associated conversion functions and macros are described in details below. For instance, here is the declaration for the C function implementing the `input` primitive:

            value input(channel, buffer, offset, length)
                    value channel, buffer, offset, length;
            {
             ...
            }

When the primitive function is applied in a Caml Light program, the C function is called with the values of the expressions to which the primitive is applied as arguments. The value returned by the function is passed back to the Caml Light program as the result of the function application.

User primitives with arity greater than 5 are implemented by C functions that receive two arguments: a pointer to an array of Caml Light values (the values for the arguments), and an integer which is the number of arguments provided:

            value prim_with_lots_of_args(argv, argn)
                    value * argv;
                    int argn;
            {
              ... argv[0] ...;              /* The first argument */
              ... argv[6] ...;              /* The seventh argument */
            }

Implementing a user primitive is actually two separate tasks: on the one hand, decoding the arguments to extract C values from the given Caml Light values, and encoding the return value as a Caml Light value; on the other hand, actually computing the result from the arguments. Except for very simple primitives, it is often preferable to have two distinct C functions to implement these two tasks. The first function actually implements the primitive, taking native C values as arguments and returning a native C value. The second function, often called the ``stub code'', is a simple wrapper around the first function that converts its arguments from Caml Light values to C values, call the first function, and convert the returned C value to Caml Light value. For instance, here is the stub code for the `input` primitive:

            value input(channel, buffer, offset, length)
                    value channel, buffer, offset, length;
            {
              return Val_long(getblock((struct channel *) channel,
                                       &Byte(buffer, Long_val(offset)),
                                       Long_val(length)));
            }

(Here, `Val_long`, `Long_val` and so on are conversion macros for the type `value`, that will be described later.) The hard work is performed by the function `getblock`, which is declared as:

            long getblock(channel, p, n)
                 struct channel * channel;
                 char * p;
                 long n;
            {
              ...
            }

To write C code that operates on Caml Light values, the following include files are provided:

Include file | Provides
--- | ---
`mlvalues.h` | definition of the `value` type, and conversion macros
`alloc.h` | allocation functions (to create structured Caml Light objects)
`memory.h` | miscellaneous memory-related functions (for in-place modification of structures, etc).

These files reside in the Caml Light standard library directory (usually `/usr/local/lib/caml-light`).

### Linking C code with Caml Light code

The Caml Light runtime system comprises three main parts: the bytecode interpreter, the memory manager, and a set of C functions that implement the primitive operations. Some bytecode instructions are provided to call these C functions, designated by their offset in a table of functions (the table of primitives).

In the default mode, the Caml Light linker produces bytecode for the standard runtime system, with a standard set of primitives. References to primitives that are not in this standard set result in the ``unavailable C primitive'' error.

In the ``custom runtime'' mode, the Caml Light linker scans the bytecode object files (`.zo` files) and determines the set of required primitives. Then, it builds a suitable runtime system, by calling the native code linker with:

- the table of the required primitives
- a library that provides the bytecode interpreter, the memory manager, and the standard primitives
- libraries and object code files (`.o` files) mentioned on the command line for the Caml Light linker, that provide implementations for the user's primitives.

This builds a runtime system with the required primitives. The Caml Light linker generates bytecode for this custom runtime system. The bytecode is appended to the end of the custom runtime system, so that it will be automatically executed when the output file (custom runtime + bytecode) is launched.

To link in ``custom runtime'' mode, execute the `camlc` command with:

- the `-custom` option
- the names of the desired Caml Light object files (`.zo` files)
- the names of the C object files and libraries (`.o` and `.a` files) that implement the required primitives. (Libraries can also be specified with the usual `-l` syntax.)

## The `value` type

All Caml Light objects are represented by the C type `value`, defined in the include file `mlvalues.h`, along with macros to manipulate values of that type. An object of type `value` is either:

- an unboxed integer
- a pointer to a block inside the heap (such as the blocks allocated through one of the `alloc_*` functions below)
- a pointer to an object outside the heap (e.g., a pointer to a block allocated by `malloc`, or to a C variable).

### Integer values

Integer values encode 31-bit signed integers. They are unboxed (unallocated).

### Blocks

Blocks in the heap are garbage-collected, and therefore have strict structure constraints. Each block includes a header containing the size of the block (in words), and the tag of the block. The tag governs how the contents of the blocks are structured. A tag lower than `No_scan_tag` indicates a structured block, containing well-formed values, which is recursively traversed by the garbage collector. A tag greater than or equal to `No_scan_tag` indicates a raw block, whose contents are not scanned by the garbage collector. For the benefits of ad-hoc polymorphic primitives such as equality and structured input-output, structured and raw blocks are further classified according to their tags as follows:

Tag | Contents of the block
--- | ---
0 to `No_scan_tag`-1 | A structured block (an array of Caml Light objects). Each field is a `value`.
`Closure_tag` | A closure representing a functional value. The first word is a pointer to a piece of bytecode, the second word is a `value` containing the environment.
`String_tag` | A character string.
`Double_tag` | A double-precision floating-point number.
`Abstract_tag` | A block representing an abstract datatype.
`Final_tag` | A block representing an abstract datatype with a ``finalization'' function, to be called when the block is deallocated.

### Pointers to outside the heap

Any pointer to outside the heap can be safely cast to and from the type `value`. This includes pointers returned by `malloc`, and pointers to C variables obtained with the `&` operator.

## Representation of Caml Light data types

This section describes how Caml Light data types are encoded in the `value` type.

### Atomic types

Caml type | Encoding
--- | ---
`int` | Unboxed integer values.
`char` | Unboxed integer values (ASCII code).
`float` | Blocks with tag `Double_tag`.
`string` | Blocks with tag `String_tag`.

### Product types

Tuples and arrays are represented by pointers to blocks, with tag 0.

Records are also represented by zero-tagged blocks. The ordering of labels in the record type declaration determines the layout of the record fields: the value associated to the label declared first is stored in field 0 of the block, the value associated to the label declared next goes in field 1, and so on.

### Concrete types

Constructed terms are represented by blocks whose tag encode the constructor. The constructors for a given concrete type are numbered from 0 to the number of constructors minus one, following the order in which they appear in the concrete type declaration. Constant constructors are represented by zero-sized blocks (atoms), tagged with the constructor number. Non-constant constructors declared with a n-tuple as argument are represented by a block of size n, tagged with the constructor number; the n fields contain the components of its tuple argument. Other non-constant constructors are represented by a block of size 1, tagged with the constructor number; the field 0 contains the value of the constructor argument. Example:

Constructed term | Representation
--- | ---
`()` | Size = 0, tag = 0
`false` | Size = 0, tag = 0
`true` | Size = 0, tag = 1
`[]` | Size = 0, tag = 0
`h::t` | Size = 2, tag = 1, first field = `h`, second field = `t`

## Operations on values

### Kind tests

- `Is_int`(v) is true if value v is an immediate integer, false otherwise
- `Is_block`(v) is true if value v is a pointer to a block, and false if it is an immediate integer.

### Operations on integers

- `Val_long`(l) returns the value encoding the `long int` l
- `Long_val`(v) returns the `long int` encoded in value v
- `Val_int`(i) returns the value encoding the `int` i
- `Int_val`(v) returns the `int` encoded in value v

### Accessing blocks

- `Wosize_val`(v) returns the size of value v, in words, excluding the header.
- `Tag_val`(v) returns the tag of value v.
- `Field`(v,n) returns the value contained in the n-th field of the structured block v. Fields are numbered from 0 to `Wosize_val`(v)-1.
- `Code_val`(v) returns the code part of the closure v.
- `Env_val`(v) returns the environment part of the closure v.
- `string_length`(v) returns the length (number of characters) of the string v.
- `Byte`(v,n) returns the n-th character of the string v, with type `char`. Characters are numbered from 0 to `string_length`(v)-1.
- `Byte_u`(v,n) returns the n-th character of the string v, with type `unsigned char`. Characters are numbered from 0 to `string_length`(v)-1.
- `String_val`(v) returns a pointer to the first byte of the string v, with type `char *`. This pointer is a valid C string: there is a null character after the last character in the string. However, Caml Light strings can contain embedded null characters, that will confuse the usual C functions over strings.
- `Double_val`(v) returns the floating-point number contained in value v, with type `double`.

The expressions `Field`(v,n), `Code_val`(v), `Env_val`(v), `Byte`(v,n), `Byte_u`(v,n) and `Double_val`(v) are valid l-values. Hence, they can be assigned to, resulting in an in-place modification of value v. Assigning directly to `Field`(v,n) must be done with care to avoid confusing the garbage collector (see below).

### Allocating blocks

From the standpoint of the allocation functions, blocks are divided according to their size as zero-sized blocks, small blocks (with size less than or equal to `Max_young_wosize`), and large blocks (with size greater than to `Max_young_wosize`). The constant `Max_young_wosize` is declared in the include file `mlvalues.h`. It is guaranteed to be at least 64 (words), so that any block with constant size less than or equal to 64 can be assumed to be small. For blocks whose size is computed at run-time, the size must be compared against `Max_young_wosize` to determine the correct allocation procedure.

- `Atom`(t) returns an ``atom'' (zero-sized block) with tag t. Zero-sized blocks are preallocated outside of the heap. It is incorrect to try and allocate a zero-sized block using the functions below. For instance, `Atom`(0) represents `()`, `false` and `[]`; `Atom`(1) represents `true`. (As a convenience, `mlvalues.h` defines the macros `Val_unit`, `Val_false` and `Val_true`.)
- `alloc`(n,t) returns a fresh small block of size n <= `Max_young_wosize` words, with tag t. If this block is a structured block (i.e. if t < `No_scan_tag`), then the fields of the block (initially containing garbage) must be initialized with legal values (using direct assignment to the fields of the block) before the next allocation.
- `alloc_tuple`(n) returns a fresh small block of size n <= `Max_young_wosize` words, with tag 0. The fields of this block must be filled with legal values before the next allocation or modification.
- `alloc_shr`(n,t) returns a fresh block of size n, with tag t. The size of the block can be greater than `Max_young_wosize`. (It can also be smaller, but in this case it is more efficient to call `alloc` instead of `alloc_shr`.) If this block is a structured block (i.e. if t < `No_scan_tag`), then the fields of the block (initially containing garbage) must be initialized with legal values (using the `initialize` function described below) before the next allocation.
- `alloc_string`(n) returns a string value of length n characters. The string initially contains garbage.
- `copy_string`(s) returns a string value containing a copy of the null-terminated C string s (a `char *`).
- `copy_double`(d) returns a floating-point value initialized with the `double` d.
- `alloc_array`(f,a) allocates an array of values, calling function f over each element of the input array a to transform it into a value. The array a is an array of pointers terminated by the null pointer. The function f receives each pointer as argument, and returns a value. The zero-tagged block returned by `alloc_array`(f,a) is filled with the values returned by the successive calls to f.
- `copy_string_array`(p) allocates an array of strings, copied from the pointer to a string array p (a `char **`).

### Raising exceptions

C functions cannot raise arbitrary exceptions. However, two functions are provided to raise two standard exceptions:

- `failwith`(s), where s is a null-terminated C string (with type `char *`), raises exception `Failure` with argument s.
- `invalid_argument`(s), where s is a null-terminated C string (with type `char *`), raises exception `Invalid_argument` with argument s.

## Living in harmony with the garbage collector

Unused blocks in the heap are automatically reclaimed by the garbage collector. This requires some cooperation from C code that manipulates heap-allocated blocks.

Rule:
  After a structured block (a block with tag less than `No_scan_tag`) is allocated, all fields of this block must be filled with well-formed values before the next allocation operation. If the block has been allocated with `alloc` or `alloc_tuple`, filling is performed by direct assignment to the fields of the block:

              Field(v, n) = vn;

  If the block has been allocated with `alloc_shr`, filling is performed through the `initialize` function:

              initialize(&Field(v, n), vn);

The next allocation can trigger a garbage collection. The garbage collector assumes that all structured blocks contain well-formed values. Newly created blocks contain random data, which generally do not represent well-formed values.

If you really need to allocate before the fields can receive their final value, first initialize with a constant value (e.g. `Val_long(0)`), then allocate, then modify the fields with the correct value (see rule 3).

Rule:
  Local variables containing values must be registered with the garbage collector (using the `Push_roots` and `Pop_roots` macros), if they are to survive a call to an allocation function.

Registration is performed with the `Push_roots` and `Pop_roots` macros. `Push_roots(`r`,`n`)` declares an array r of n values and registers them with the garbage collector. The values contained in r`[0]` to r`[`n`-1]` are treated like roots by the garbage collector. A root value has the following properties: if it points to a heap-allocated block, this block (and its contents) will not be reclaimed; moreover, if this block is relocated by the garbage collector, the root value is updated to point to the new location for the block. `Push_roots(`r`,`n`)` must occur in a C block exactly between the last local variable declaration and the first statement in the block. To un-register the roots, `Pop_roots()` must be called before the C block containing `Push_roots(`r`,`n`)` is exited. (Roots are automatically un-registered if a Caml exception is raised.)

Rule:
  Direct assignment to a field of a block, as in

              Field(v, n) = w;

  is safe only if v is a block newly allocated by `alloc` or `alloc_tuple`; that is, if no allocation took place between the allocation of v and the assignment to the field. In all other cases, never assign directly. If the block has just been allocated by `alloc_shr`, use `initialize` to assign a value to a field for the first time:

              initialize(&Field(v, n), w);

  Otherwise, you are updating a field that previously contained a well-formed value; then, call the `modify` function:

              modify(&Field(v, n), w);

To illustrate the rules above, here is a C function that builds and returns a list containing the two integers given as parameters:

    value alloc_list_int(i1, i2)
            int i1, i2;
    {
      value result;
      Push_roots(r, 1);
      r[0] = alloc(2, 1);                     /* Allocate a cons cell */
      Field(r[0], 0) = Val_int(i2);           /* car = the integer i2 */
      Field(r[0], 1) = Atom(0);               /* cdr = the empty list [] */
      result = alloc(2, 1);                   /* Allocate the other cons cell */
      Field(result, 0) = Val_int(i1);         /* car = the integer i1 */
      Field(result, 1) = r[0];                /* cdr = the first cons cell */
      Pop_roots();
      return result;
    }

The ``cons'' cell allocated first needs to survive the allocation of the other cons cell; hence, the value returned by the first call to `alloc` must be stored in a registered root. The value returned by the second call to `alloc` can reside in the un-registered local variable `result`, since we won't do any further allocation in this function.

In the example above, the list is built bottom-up. Here is an alternate way, that proceeds top-down. It is less efficient, but illustrates the use of `modify`.

    value alloc_list_int(i1, i2)
            int i1, i2;
    {
      value tail;
      Push_roots(r, 1);
      r[0] = alloc(2, 1);                     /* Allocate a cons cell */
      Field(r[0], 0) = Val_int(i1);           /* car = the integer i1 */
      Field(r[0], 1) = Val_int(0);            /* A dummy value
      tail = alloc(2, 1);                     /* Allocate the other cons cell */
      Field(tail, 0) = Val_int(i2);           /* car = the integer i2 */
      Field(tail, 1) = Atom(0);               /* cdr = the empty list [] */
      modify(&Field(r[0], 1), tail);          /* cdr of the result = tail */
      Pop_roots();
      return r[0];
    }

It would be incorrect to perform `Field(r[0], 1) = tail` directly, because the allocation of `tail` has taken place since `r[0]` was allocated.

## A complete example

This section outlines how the functions from the Unix `curses` library can be made available to Caml Light programs. First of all, here is the interface `curses.mli` that declares the `curses` primitives and data types:

    type window;;                   (* The type "window" remains abstract *)
    value initscr: unit -> window = 1 "curses_initscr"
      and endwin: unit -> unit = 1 "curses_endwin"
      and refresh: unit -> unit = 1 "curses_refresh"
      and wrefresh : window -> unit = 1 "curses_wrefresh"
      and newwin: int -> int -> int -> int -> window = 4 "curses_newwin"
      and mvwin: window -> int -> int -> unit = 3 "curses_mvwin"
      and addch: char -> unit = 1 "curses_addch"
      and mvwaddch: window -> int -> int -> char -> unit = 4 "curses_mvwaddch"
      and addstr: string -> unit = 1 "curses_addstr"
      and mvwaddstr: window -> int -> int -> string -> unit = 4 "curses_mvwaddstr"
    ;; (* lots more omitted *)

To compile this interface:

            camlc -c curses.mli

To implement these functions, we just have to provide the stub code; the core functions are already implemented in the `curses` library. The stub code file, `curses.o`, looks like:

    #include <curses.h>
    #include <mlvalues.h>

    value curses_initscr(unit)
            value unit;
    {
      return (value) initscr();     /* OK to coerce directly from WINDOW * to value
                                       since that's a block created by malloc() */
    }

    value curses_wrefresh(win)
            value win;
    {
      wrefresh((WINDOW *) win);
      return Val_unit;
    }

    value curses_newwin(nlines, ncols, x0, y0)
            value nlines, ncols, x0, y0;
    {
      return (value) newwin(Int_val(nlines), Int_val(ncols),
                            Int_val(x0), Int_val(y0));
    }

    value curses_addch(c)
            value c;
    {
      addch(Int_val(c));            /* Characters are encoded like integers */
      return Val_unit;
    }

    value curses_addstr(s)
            value s;
    {
      addstr(String_val(s));
      return Val_unit;
    }

    /* This goes on for pages. */

(Actually, it would be better to create a library for the stub code, with each stub code function in a separate file, so that linking would pick only those functions from the `curses` library that are actually used.)

The file `curses.c` can be compiled with:

            cc -c -I/usr/local/lib/caml-light curses.c

or, even simpler,

            camlc -c curses.c

(When passed a `.c` file, the `camlc` command simply calls `cc` on that file, with the right `-I` option.)

Now, here is a sample Caml Light program `test.ml` that uses the `curses` module:

    #open "curses";;
    let main_window = initscr () in
    let small_window = newwin 10 5 20 10 in
      mvwaddstr main_window 10 2 "Hello";
      mvwaddstr small_window 4 3 "world";
      refresh();
      for i = 1 to 100000 do () done;
      endwin()
    ;;

To compile this program, run:

            camlc -c test.ml

Finally, to link everything together:

            camlc -custom -o test test.zo curses.o -lcurses