День: 06.06.2018

Hello readers,

I know this time it is a little late, but I am also busy with some other professional things. 🙂

This time let’s discuss about the process creation in OpenBSD operating system from user-space level to kernel space.

We will take an example of the user-space process that will be launched from the Command Line Interface (console), for example, “ls”, and then what happens in kernel-space as a result of it.

I will divide this series into 3 parts, like 

, 

, 

, because the creation of process itself took some amount of time for me to learn, and analyzing or tracking from user-space to kernel-space had to be done line by line.

I have used gdb to debug the process and analyze it line by line.

Now, I will not waste your time too much.

Let’s dive into the user-space to kernel-space and learn and see the beauty of puffer.

I have divided the full process and functions that are used in the kernel into the points, so, I think it will be easy to read and learn.

Now, suppose you have launched “ls” command from CLI (xterm):

Here, the 

 process is “ksh”, that is, default shell in OpenBSD which invokes “ls” command or any other command.

Every process is created by 

 , that is, fork system call which is indirectly (internally) calls 

() creates a new process out of p1, which should be the current thread. This function is used primarily to implement the fork(2) and vfork(2) system calls, as well as the kthread_create(9) function.

Life cycle of a process (in brief):

“ls” → fork(2) → sys_fork() → fork1() → sys_execve() → sys_exit() → exit1()

Under the hood working of 

fork1

()

After “ls” from user-space it goes to 

() (libc) then from there to 

().

: It is a macro which defines that the call is done by the fork(2)system call. Used only for statistics.

• So, the value of 
  flags

   variable is set to 

  1

   , because the call is done by 

  fork(2)

  .

• check for PTRACING then update the 
  flags

   with 

  PTRACE_FORK

   else leave it and return to the 

  fork1()

Now, 

• The above code includes, 
  curp->p_p->ps_comm

   is “ksh”, that is, parent process which will fork “ls” (user-space).

• Initially some process structures, then, setting
  uid = curp->p_ucred->cr_ruid

   , it means setting the uid as real user id.

• Then, the structure for process address space information.
• Then, some variables and 
  ptrace_state

   structure and then the condition checking using 

  KASSERT

  .

• fork_check_maxthread(uid)

   → it is used to the check or track the number of threads invoked by the specific 

  uid

   .

• It checks the number of threads invoked by specific 
  uid

   shouldn’t be greater than the number of maximum threads allowed or also for 

  maxthread —5

   . Because the last 5 process from the 

  maxthread

   is reserved for the root.

• If it is greater than defined 
  maxthread

   or 

  maxthread — 5

  , it will print the message

  tablefull

  once every 10 seconds. Else, it will increment the number of threads.

• Now, after 
  fork_check_thread

  , again, the same implementation happens for tracking process. If you want you can have a look in our 

  fork1

   code screen-shot.

Now, we will proceed further,

• It is changing the count of threads for a specific user via 
  chgproccnt(uid,1)

  .

• uidinfo

   structure maintains every uid resource consumption counts, including the process count and socket buffer space usage.

• uid_find

   function looks up and returns the 

  uidinfo

   structure for 

  <em class="markup--em markup--li-em">uid</em>

  . If no 

  uidinfo

   structure exists for 

  <em class="markup--em markup--li-em">uid</em>

  , a new structure will be allocated and initialized.

Then, it increments the 

 , that is, number of processes by 

and then returns count.

After, that, it is checks for the non-privileged 

 and also that the number of process is greater than the soft limit of resources, that is, 

, from what I have found in gdb.

Have a look in the below screen-shot for the proper view of values:

If non-privileged is allowed and the count is increased by the maximum resource limit, it will decrease the count via 

 by passing 

 as 

 parameter and also decrease the number of processes and threads.

• Next, the 
  <a class="markup--anchor markup--li-anchor" href="https://man.openbsd.org/uvm.9" target="_blank" rel="nofollow noopener" data-href="https://man.openbsd.org/uvm.9">uvm_uarea_alloc</a>()

   function allocates a thread’s ‘uarea’, the memory where its kernel stack and PCB are stored.

Now, it checks if the 

 variable doesn’t contain any thread’s address, if it is zero, then it decrements the count of the number of process and thread.

Now, there are the some important functions:

→ 

→ 

Here, in the 

 function, we will get our user-space process, that is, in our case “ls”. The process gets retrieved from the pool of process, that is, 

 via 

 function.

Then, we set the state of the thread to be 

 , which means that the process/thread is being created by 

 . We then set

Now, they are zeroing the section of 

 . See, the below code snippet from 

In above code snippet, all the variables will be zeroed via 

 upon creation in the fork.

Then, they are copying the section from 

 to

via 

. Have a look below in the screen-shot to know which of the field members will be copied.

• The, 
  crhold(p-&gt;p_ucred)

   means it will increment the reference count in 

  struct ucred

   structure, that is, 

  p-&gt;p_ucred-&gt;cr_ref++

   .

• Now, typecast the thread’s addr, that is, 
  (struct user *)uaddr

   and save it in kernel’s virtual addr of u-area.

• Now, it will initialize the timeout.

dummy function to show the 

 function working.

timeout_set(timeout, b, argument)

It means initialize the 

 struture and call the function 

 with 

 .

void
timeout_set(struct timeout *new, void (*fn)(void *), void *arg)
{
        new->to_func = fn;
        new->to_arg = arg;
        new->to_flags = TIMEOUT_INITIALIZED;
}

scheduler_fork_hook(parent, p): It is a macro which will update the 

 of child from parent’s 

.

 holds an estimate of the amount of CPU that the process has used recently

/* Inherit the parent’s scheduler history */
#define scheduler_fork_hook(parent, child) do {    \
 (child)->p_estcpu = (parent)->p_estcpu;           \
} while (0)

Then, return the newly created thread 

 .

Now, another important function is 

 which will create the process in a similar fashion to what we have seen above in the 

func.

• process_new(struct proc *p, struct process *parent, int flags)

In above code snippet, the same thing is happening again like select process from 

 via 

 then zeroing using 

 and copying using 

.

So, for the detailed explanation, please go through the 

 function first.

Next is initialization of process using 

 function.

 : It is the original and main thread in the process. It’s only special for the handling of 

 and some signal and ptrace behaviours that need to be fixed.

→Copy initial thread, that is, 

 to 

 .

→Initialize the queue with referenced by head. Here, head is 

. Then, Insert 

 at the TAIL of the queue. Here, elm is 

 .

→set the number of references to 

, that is, 

→copy the process 

 to the process of initial thread.

→set the same creds for process as the initial thread.

→condition check for the new thread and the new process via 

.

→Initialize the List referenced by head. Here, head is 

→Again, initialize timeout. (for detail, see 

)

Now, after the process initialization, pid allocation takes place.

 

 returns unused pid

 internally calls the 

 which again calls the 

 then via 

 a fully randomized number is returned which is used as pid.

Then, for the availability of pid, or in other words, for unused pid, it verifies that whether the new pid is already taken or not by any process. It verifies this one by one in the process, process groups, and zombie process by using function 

which internally calls these functions:

• prfind(pid_t pid) : Locate a process by number
• pgfind(pid_t pgid) : Locate a process group by number
• zombiefind(pid_t pid :Locate a zombie process by number

Now, store the pointer to parent process in 

 .

Increment the number of references count in process limit structure, that is, 

 .

Store the vnode of executable of parent into 

 ,that is, 

 .

if (pr→ps_textvp)
        vref(pr→ps_textvp); /* vref --> vnode reference */

Above code snippet means, if valid vnode found then increment the 

 variable inside the 

 structure of the executable.

Now, the calculation for setting up process flags:

pr→ps_flags = parent →ps_flags & (PS_SUGID | PS_SUGIDEXEC | PS_PLEDGE | PS_EXECPLEDGE | PS_WXNEEDED);

pr →ps_flags = parent →ps_flags & (0x10 | 0x20 | 0x100000 | 0x400000 | 0x200000)

if (vnode of controlling terminal != NULL)

 

pr→ps_flags |= parent→ps_flags & PS_CONTROLT;

 continued…

Checks:

* if child_able_to_share_file_descriptor_table_with_parent:
         pr->ps_fd = fdshare(parent)      /* share the table */
  else
         pr->ps_fd = fdcopy(parent)       /* copy the table */

* if child_able_to_share_the_parent's_signal_actions:
         pr->ps_sigacts = sigactsshare(parent) /* share */
  else
         pr->ps_sigacts = sigactsinit(parent)  /* copy */

* if child_able_to_share_the_parent's addr space:
         pr->ps_vmspace = uvmspace_share(parent)
  else
         pr->ps_vmspace = uvmspace_fork(parent)

* if process_able_to_start_profiling:
         smartprofclock(pr);    /* start profiling on a process */

* if check_child_able_to_start_ptracing:
         pr->ps_flags |= parent->ps_flags & PS_PTRACED

* if check_no_signal_or_zombie_at_exit:
         pr->ps_flags |= PS_NOZOMBIE /*No signal or zombie at exit

* if check_signals_stat_swaping:
         pr->ps_flags |= PS_SYSTEM

update the 

 with PS_EMBRYO by ORing it, that is,

 /* New process, not yet fledged */

 → Force visibility of all of the above changes.

— All stores preceding the memory barrier will reach global visibility before any stores after the memory barrier reach global visibility.

In short, I think it is used to forcefully make visible changes globally.

Now, Insert the new 

, that is, 

 at the head of the list. Here, head is 

 .

• return 
  pr

fork1()

 continued…

Substructures

 and 

 directly copy of 

 and 

.

checks,

** if (process_has_no_signals_stats_or_swapping) then atomically set bits.

** if (child_is_suspending_the_parent_process_until_the_child_is terminated (by calling _exit(2) or abnormally), or makes a call to execve(2)) then atomically set bits,

atomic_setbits_int(pr →ps_flags, PS_PPWAIT);
atomic_setbits_int(pr →ps_flags, PS_ISPWAIT);

#ifdef KTRACE
/* Some KTRACE related things */
#endif

cpu_fork(curp, p, NULL, NULL, func, arg ?arg: p)

— To create or Update PCB and make child ready to RUN.

/*
 * Finish creating the child thread. cpu_fork() will copy
 * and update the pcb and make the child ready to run. The
 * child will exit directly to user mode via child_return()
 * on its first time slice and will not return here.
 */

Address space,

if (call is done by fork syscall); then
increment the number of fork() system calls.
update the vm_pages affected by fork() syscall with addition of data page and stack page.

else if (call is done by vfork() syscall); then
do as same as if it was fork syscall but for vfork system call. (see above if {for fork})

else
increment the number of kernel threads created.

Check,

If (process is being traced && created by fork system call);then
{
        The malloc() function allocates the uninitialized memory in the kernel address space for an object whose size is specified by size, that is, here, sizeof(*newptstat). And,

<em class="markup--em markup--pre-em">struct ptrace_state *newptstat</em>

}

allocate thread ID, that is, 

This is also the same calling 

 directly and using 

 function for finding the thread ID by number.

* inserts the new element p at the head	of the allprocess list.
* insert the new element p at the head of the thread hash list.
* insert the new element pr at the head of the process hash list.
* insert the new element pr after the curpr element.
* insert the new element pr at the head of the children process  list.

fork1()

 continued…

Again,
If (isProcessPTRACED())
{
then save the parent process id during ptracing, that is,

 .
If (pointer to parent process_of_child != pointer to parent process_of_current_process)
{
proc_reparent(pr, curpr→ps_pptr); /* Make current process the new parent of process child, that is, 

*/

Now, check whether 

contains some address, in our case, 

 contains a kernel virtual address returned by 

.
If above condition is 

, that is, 

 . Then, set the ptrace status:
Set 

 point to the ptrace state structure. Then, make the 

point to 

 .

→Update the ptrace status to the 

 process and also the 

 process.

curpr->ps_ptstat->pe_report_event = PTRACE_FORK;
pr->ps_ptstat->pe_report_event = PTRACE_FORK;
curpr->ps_ptstat->pe_other_pid = pr->ps_pid;
pr->ps_ptstat->pe_other_pid = curpr->ps_pid;

Now, for the new process set accounting bits and mark it as complete.

• get the nano time to start the process.
• Set accounting flags to 
  AFORK

   which means forked but not execed.

• atomically clear the bits.
• Then, check for the new child is in the IDLE state or not, if yes then make it runnable and add it to the run queue by 
  fork_thread_start

   function.

• If it is not in the IDLE state then put 
  arg

   to the current CPU, running on.

Freeing the memory or kernel virtual address that is allocated by malloc for 

 via 

 .

Notify any interested parties about the new process via 

 .

Now, update the stats counter for successfully forked.

uvmexp.forks++; /* -->For forks */

if (flags & FORK_PPWAIT)
        uvmexp.forks_ppwait++; /* --> counter for forks where parent waits */

if (flags & FORK_SHAREVM)
        uvmexp.forks_sharevm++; /* --> counter for forks where vmspace is shared */

Now, pass pointer to the new process to the caller.

if (rnewprocp != NULL)
        *rnewprocp = p;

• setting the 
  PPWAIT

   on child and the 

  PS_ISPWAIT

   on ourselves, that is, the parent and then go to the sleep on our process via 

  tsleep

   .

• Check, If the child is started with tracing enables && the current process is being traced then alert the parent by using 
  SIGTRAP

   signal.

• Now, return the child pid to the parent process.
• <strong class="markup--strong markup--li-strong"><em class="markup--em markup--li-em">return (0)</em></strong>

Then, finally, I have seen in the debugger that after the 

, it jumps to 

 file for system call handling and for the setting frame.

Some of the machine independent (MI) functions defined in 

 file, like, 

, 

 and 

.

Then, after handling the system calls from 

 then, control pass to the 

 system call, which I will explain later (in the second part) and also I will explain more about the 

code in upcoming posts. It has already become a long post.

References:

• OpenBSD Source Codes
• OpenBSD kernel Internals — The Hitchhiker’s Guide
• OpenBSD manual pages.
• BSD Virtual Memory
• NetBSD manual pages.
• FreeBSD manual pages.
• Understanding The Linux Kernel
• Linux Kernel Development — Robert Love
• Google 🙂