Fork protocol

The OpenMP parallel section pragma is translated by the compiler into a set of forking and register transmission instructions forming the forking protocol.
The forking protocol is a set of instructions to prepare, send and receive:

the forked hart,

the join hart,

the parallelizing call indication,

the continuation values if any.

The forked hart (or next hart) is the hart allocated by the p_fn or p_fc instruction (the p_fc instruction allocates a new hart on the same core and the p_fn allocates a new hart on the next core). The allocated hart identification is saved in the p_fn or p_fc destination register (usually, register t6; the two least significant bits are the allocated hart number and the most significant bits are the core number hosting the allocated hart).

The join hart is the hart where the next sequential continuation resides. The join hart identification is saved with a p_set instruction. The p_set instruction writes the current hart identification in the upper half of its destination register (usually, register t0; bits 16-17 for the join hart number and bits 18-30 for the join core number).

The next hart and the join hart are merged in a single register with the p_merge instruction (usually, register t0; the upper half holds the join hart identification and the lower half holds the next hart identification). Bit 31 of the p_merge destination register is set to indicate that the next call is parallelized (parallelizing call indication).

In the example, the main function sets register t0 with value -1 to indicate that there is no successor to the current hart and no join hart. The main ending return (instruction p_jalr zero, ra, t0 just before par1 label) will exit as t0=-1 (we assume the main function return ends the processor run rather than an OS process; it is an exit rather than a return to the standard _start Unix function).
The main function saves the return address and the join hart on the stack.
The main function calls function par1 (added by the compiler from the pragma omp parallel). The return address saved in register ra is the join label.
Hart 0 waits for a return to label join which will be sent by the return from the parallel block in hart 2.

In par1 register t0 is set with the current hart identification (instruction p_set t0, t0; the current hart is number 0). This is the join hart initialization which will be transmitted to function f.
Then, the code forks, i.e. allocates a new hart on the same core with instruction p_fc t6. The allocated hart identification is saved in the destination register t6 (hart number 1).
Registers ra and t0 are transmitted to the allocated hart with p_swcv instructions (constants 0 and 4 in the p_swcv instructions are the offsets in the local memory used to store the transferred values). Register ra contains the join address (i.e. join) and register t0 contains the join hart identification (i.e. hart 0).
The allocated hart identification is merged with the join hart in register t0 (instruction p_merge t0, t0, t6; t0 upper half keeps the join hart identification and t0 lower half is set to the allocated hart, which is the new next hart, i.e. hart 1).
A memory synchronization instruction (p_syncm) is inserted between the last p_swcv and the p_jal to ensure that all the sendings are done before their reception starts.
The p_jal instruction sends the return address .L1 to the allocated hart and clears register ra to indicate that there is no local return.
The called address, i.e. function f, is run locally and the continuation after return, i.e. label .L1, is run on the allocated hart.

The same forking protocol is applied to g (with a p_fn which allocates a hart on the next core instead of a p_fc). Function f is run on hart 0, function g is run on hart 1 and the end of the parallel block is run on hart 2. Hart 0 has hart 1 as its successor. Hart 1 has hart 2 as its successor. Hart 2 has no successor.

When f returns, as register ra is 0, there is no return point. As register t0 indicates that the join hart is hart 0, the current hart has to wait for a resume address (join hart==current hart means wait until the resume address is received). Hart 0 is stopped. It sends a join signal to its successor (hart 1). Hart 1 may not end before it has received the join signal from its predecessor, which ensures harts are ended in order.

When g returns, as register ra is 0, there is no return point. As register t0 indicates that the join hart is hart 0 (i.e. not the current hart), hart 1 ends (join hart!=current hart means end). It sends its join signal to its successor (hart 2).

When hart 2 returns, as register ra is not 0, it is a join address. Register t0 contains the join hart identification. Hence, the join address join is sent to hart 0 which is resumed. Hart 2 is ended (join hart!=current hart). As it has no successor, it sends no join signal. It sends its successor (none or -1) as the new hart 0 successor.

At the return and join point, the main function restores the return address and join hart before exiting with the p_jalr instruction (ra==0 and join hart==-1 means exit).