IHK/McKernel

IHK/McKernel is a light-weight multi-kernel operating system designed for high-end supercomputing. It runs Linux and McKernel, a light-weight kernel (LWK), side-by-side inside compute nodes and aims at the following:

• Provide scalable and consistent execution of large-scale parallel scientific applications, but at the same time maintain the ability to rapidly adapt to new hardware features and emerging programming models

• Provide efficient memory and device management so that resource contention and data movement are minimized at the system level

• Eliminate OS noise by isolating OS services in Linux and provide jitter free execution on the LWK

• Support the full POSIX/Linux APIs by selectively offloading (slow-path) system calls to Linux

See Quick Guide – Installation for jump start.

1. Introduction

IHK/McKernel is a light-weight multi-kernel operating system designed for high-end supercomputing. It runs Linux and McKernel, a light-weight kernel (LWK), side-by-side inside compute nodes and aims at the following:

• Provide scalable and consistent execution of large-scale parallel scientific applications, but at the same time maintain the ability to rapidly adapt to new hardware features and emerging programming models

• Provide efficient memory and device management so that resource contention and data movement are minimized at the system level

• Eliminate OS noise by isolating OS services in Linux and provide jitter free execution on the LWK

• Support the full POSIX/Linux APIs by selectively offloading (slow-path) system calls to Linux

2. Background and Motivation

With the growing complexity of high-end supercomputers, the current system software stack faces significant challenges as we move forward to exascale and beyond. The necessity to deal with extreme degree of parallelism, heterogeneous architectures, multiple levels of memory hierarchy, power constraints, etc., advocates operating systems that can rapidly adapt to new hardware requirements, and that can support novel programming paradigms and runtime systems. On the other hand, a new class of more dynamic and complex applications are also on the horizon, with an increasing demand for application constructs such as in-situ analysis, workflows, elaborate monitoring and performance tools. This complexity relies not only on the rich features of POSIX, but also on the Linux APIs (such as the /proc, /sys filesystems, etc.) in particular.

2.1. Two Traditional HPC OS Approaches

Traditionally, light-weight operating systems specialized for HPC followed two approaches to tackle scalable execution of large-scale applications. In the full weight kernel (FWK) approach, a full Linux environment is taken as the basis, and features that inhibit attaining HPC scalability are removed, i.e., making it light-weight. The pure light-weight kernel (LWK) approach, on the other hand, starts from scratch and effort is undertaken to add sufficient functionality so that it provides a familiar API, typically something close to that of a general purpose OS, while at the same time it retains the desired scalability and reliability attributes. Neither of these approaches yields a fully Linux compatible environment.

2.2. The Multi-kernel Approach

A hybrid approach recognized recently by the system software community is to run Linux simultaneously with a lightweight kernel on compute nodes and multiple research projects are now pursuing this direction. The basic idea is that simulations run on an HPC tailored lightweight kernel, ensuring the necessary isolation for noiseless execution of parallel applications, but Linux is leveraged so that the full POSIX API is supported. Additionally, the small code base of the LWK can also facilitate rapid prototyping for new, exotic hardware features. Nevertheless, the questions of how to share node resources between the two types of kernels, where do device drivers execute, how exactly do the two kernels interact with each other and to what extent are they integrated, remain subjects of ongoing debate.

3. Architectural Overview

At the heart of the stack is a low-level software infrastructure called Interface for Heterogeneous Kernels (IHK). IHK is a general framework that provides capabilities for partitioning resources in a many-core environment (e.g.,CPU cores and physical memory) and it enables management of lightweight kernels. IHK can allocate and release host resources dynamically and no reboot of the host machine is required when altering configuration. IHK also provides a low-level inter-kernel messaging infrastructure, called the Inter-Kernel Communication (IKC) layer. An architectural overview of the main system components is shown below.

McKernel is a lightweight kernel written from scratch. It is designed for HPC and is booted from IHK. McKernel retains a binary compatible ABI with Linux, however, it implements only a small set of performance sensitive system calls and the rest are offloaded to Linux. Specifically, McKernel has its own memory management, it supports processes and multi-threading with a simple round-robin cooperative (tick-less) scheduler, and it implements signaling. It also allows inter-process memory mappings and it provides interfaces to hardware performance counters.

3.1. Functionality

An overview of some of the principal functionalities of the IHK/McKernel stack is provided below.

3.1.1. System Call Offloading

System call forwarding in McKernel is implemented as follows. When an offloaded system call occurs, McKernel marshals the system call number along with its arguments and sends a message to Linux via a dedicated IKC channel. The corresponding proxy process running on Linux is by default waiting for system call requests through an ioctl() call into IHK’s system call delegator kernel module. The delegator kernel module’s IKC interrupt handler wakes up the proxy process, which returns to userspace and simply invokes the requested system call. Once it obtains the return value, it instructs the delegator module to send the result back to McKernel, which subsequently passes the value to user-space.

3.1.2. Unified Address Space

The unified address space model in IHK/McKernel ensures that offloaded system calls can seamlessly resolve arguments even in case of pointers. This mechanism is depicted below and is implemented as follows.

First, the proxy process is compiled as a position independent binary, which enables us to map the code and data segments specific to the proxy process to an address range which is explicitly excluded from McKernel’s user space. The grey box on the right side of the figure demonstrates the excluded region. Second, the entire valid virtual address range of McKernel’s application user-space is covered by a special mapping in the proxy process for which we use a pseudo file mapping in Linux. This mapping is indicated by the blue box on the left side of the figure.

4. Installation

The following OS distributions and platforms are recommended:

• OS distribution

  • CentOS 7.3 or later

  • RHEL 7.3 or later

• Platform

  • Intel Xeon

  • Intel Xeon Phi

  • Fujitsu A64FX

4.1. Prepare files for building McKernel

Grant read permission to the System.map file of your kernel version on the build machine:

sudo chmod a+r /boot/System.map-`uname -r`

Install the following packages to the build machine:

cmake kernel-devel binutils-devel systemd-devel numactl-devel gcc make nasm git libdwarf-devel capstone-devel

4.1.1. When having access to repositories

On RHEL 8, enable the CodeReady Linux Builder (CLB) repository:

sudo subscription-manager repos --enable codeready-builder-for-rhel-8-$(/bin/arch)-rpms

On CentOS 8, enable the PowerTools repository:

sudo dnf config-manager --set-enabled PowerTools

Enable EPEL repository:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

Install with yum:

sudo yum install cmake kernel-devel binutils-devel systemd-devel numactl-devel gcc make nasm git libdwarf-devel capstone-devel

4.1.2. When not having access to repositories

libdwarf-devel

Ask the system administrator to install them. Note that libdwarf-devel is in the CodeReady Linux Builder repository on RHEL 8 or in the PowerTools repository on CentOS 8.

capstone-devel

1. Ask the system administrator to install capstone-devel. Note that it is in the EPEL repository.

2. Download the rpm with the machine in which you are the administrator:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo yum install yum-utils
yumdownloader capstone-devel

And then install it to your home directory:

cd $HOME/$(uname -p)
rpm2cpio capstone-devel-4.0.1-9.el8.aarch64.rpm | cpio -idv
sed -i 's#/usr/#'"$HOME"'/'"$(uname -p)"'/usr/#' $HOME/$(uname -p)/usr/lib64/pkgconfig/capstone.pc

4.2. Clone, compile, install

Clone the source code:

mkdir -p ~/src/ihk+mckernel/
cd ~/src/ihk+mckernel/
git clone --recursive -b development https://github.com/ihkmckernel/mckernel.git

(Optional) Checkout to the specific branch or version:

cd mckernel
git checkout <pathspec>
git submodule update

Foe example, if you want to try the development branch, use “development” as the pathspec. If you want to try the prerelease version 1.7.0-0.2, use “1.7.0-0.2”.

Move to build directory:

mkdir -p ~/src/ihk+mckernel/build && cd ~/src/ihk+mckernel/build

Run cmake:

4.2.1. When not cross-compiling:

CMAKE_PREFIX_PATH=${HOME}/$(uname -p)/usr \
  cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel \
  -DENABLE_UTI=ON \
  ../mckernel

Note that CMAKE_PREFIX_PATH=${HOME}/$(uname -p)/usr is required only when capstone-devel is installed to your home directory.

4.2.2. When cross-compiling:

cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel \
  -DUNAME_R=<target_uname_r> \
  -DKERNEL_DIR=<kernnel_dir> \
  -DBUILD_TARGET=smp-arm64 \
  -DCMAKE_TOOLCHAIN_FILE=../mckernel/cmake/cross-aarch64.cmake \
  -DENABLE_UTI=ON \
  ../mckernel

4.2.3. Install with cmake

Install with make:

make -j install

The kernel modules and McKernel kernel image should be installed under the ihk+mckernel folder in your home directory.

4.2.4. Install with rpm

Create the tarball and the spec file:

make dist
cp mckernel-<version>.tar.gz <rpmbuild>/SOURCES

Create the rpm package:

When not cross-compiling:

Then build the rpm:

rpmbuild -ba scripts/mckernel.spec

When cross-compiling:

rpmbuild -ba scripts/mckernel.spec --target <target_uname_m> -D 'kernel_version <target_uname_r>' -D 'kernel_dir <kernel_source>'

Install the rpm package:

sudo rpm -ivh <rpmbuild>/RPMS/<arch>/mckernel-<version>-<release>_<linux_kernel_ver>_<dist>.<arch>.rpm

The kernel modules and McKernel kernel image are installed under the standard system directories.

4.3. Prepare files and change settings for installing McKernel

Disable SELinux of the compute nodes:

sudo vim /etc/selinux/config

Change the file to SELINUX=disabled. And then reboot the compute nodes:

sudo reboot

Install the following packages to the compute nodes:

systemd-libs numactl-libs libdwarf capstone

4.3.1. When having access to repositories

On RHEL 8, enable the CodeReady Linux Builder (CLB) repository:

sudo subscription-manager repos --enable codeready-builder-for-rhel-8-$(/bin/arch)-rpms

On CentOS 8, enable the PowerTools repository:

sudo dnf config-manager --set-enabled PowerTools

Enable EPEL repository:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

Install with yum:

sudo yum install systemd-libs numactl-libs libdwarf capstone

4.3.2. When not having access to repositories

libdwarf

Ask the system administrator to install them. Note that libdwarf is in the CodeReady Linux Builder repository on RHEL 8 or in the PowerTools repository on CentOS 8.

capstone

1. Ask the system administrator to install capstone. Note that it is in the EPEL repository.

2. Download the rpm with the machine in which you are the administrator:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo yum install yum-utils
yumdownloader capstone

and then install it to your home directory:

cd $HOME/$(uname -p)
rpm2cpio capstone-4.0.1-9.el8.aarch64.rpm | cpio -idv

4.4. Boot McKernel

A boot script called mcreboot.sh is provided under sbin in the install folder. To boot on logical CPU 1 with 512MB of memory, use the following invocation:

export TOP=${HOME}/ihk+mckernel/
cd ${TOP}
sudo ./sbin/mcreboot.sh -c 1 -m 512m

You should see something similar like this if you display the McKernel’s kernel message log:

./sbin/ihkosctl 0 kmsg

IHK/McKernel started.
[ -1]: no_execute_available: 1
[ -1]: map_fixed: phys: 0xfee00000 => 0xffff860000009000 (1 pages)
[ -1]: setup_x86 done.
[ -1]: ns_per_tsc: 385
[ -1]: KCommand Line: hidos    dump_level=24
[ -1]: Physical memory: 0x1ad3000 - 0x21000000, 525520896 bytes, 128301 pages available @ NUMA: 0
[ -1]: NUMA: 0, Linux NUMA: 0, type: 1, available bytes: 525520896, pages: 128301
[ -1]: NUMA 0 distances: 0 (10),
[ -1]: map_fixed: phys: 0x28000 => 0xffff86000000a000 (2 pages)
[ -1]: Trampoline area: 0x28000
[ -1]: map_fixed: phys: 0x0 => 0xffff86000000c000 (1 pages)
[ -1]: # of cpus : 1
[ -1]: locals = ffff880001af6000
[  0]: BSP: 0 (HW ID: 1 @ NUMA 0)
[  0]: BSP: booted 0 AP CPUs
[  0]: Master channel init acked.
[  0]: vdso is enabled
IHK/McKernel booted.

4.5. Run a simple program on McKernel

The mcexec command line tool (which is also the Linux proxy process) can be used for executing applications on McKernel:

./bin/mcexec hostname
centos-vm

4.6. Shutdown McKernel

Finally, to shutdown McKernel and release CPU/memory resources back to Linux use the following command:

sudo ./sbin/mcstop+release.sh

5. The Team

The McKernel project was started at The University of Tokyo and currently it is mainly developed at RIKEN. Some of our collaborators include:

• Hitachi

• Fujitsu

• CEA (France)

• NEC

6. License

McKernel is GPL licensed, as found in the LICENSE file.

7. Contact

Please give your feedback to us via the following mailing list: ihkmckernel@googlegroups.com

1. Architectural Overview

See Quick Guide – Architectural Overview.

2. Running Programs

You need to check if your application and pre-/post-processing programs are suited to run with McKernel. Follow the guide below to choose to run the whole on McKernel, or run the whole on Linux, or run pre-/post-processing on Linux and the application on McKernel:

• Application

  • Run the whole on Linux if it issues system calls frequently and becoming the bottleneck with McKernel, e.g., those performing many file I/O operations.

  • Otherwise, run it on McKernel.

• Pre-/Post-processing

  • Run it on McKernel if it consumes a large amount of memory or the execution time isn’t prolonged prohivitively with McKernel. The reason for the first condition is that the resource could be limited for Linux CPUs in the nodes for McKernel.

  • Otherwise, run it on Linux.

2.1. Modify job script

When using job submission system, you need to modify the job scripts so that the job script itself is going to run on Linux. For example, with Fujitsu Technical Computing Suite (TCS), you need to specify jobenv=mck1 by inserting the following line into the job script:

#PJM -L jobenv=mck1

2.2. (Optional, Fujitsu TCS only) Specify boot parameters

You can specify the boot parameters by defining environmental variables and pass them to Fujitsu TCS. The parameters include the resource reservation settings, resource reservation amount, kernel arguments and routing of message channels between McKernel CPUs and Linux CPUs. See IHK Specifications - ihk_create_os_str() for the parameter names and allowed values. The example of setting the memory amount is shown below.

export IHK_MEM="7G@4,7G@5,7G@6,7G@7"
pjsub -X run.sh

2.3. Insert mcexec into the command line

You need to insert mcexec into the command lines invoking the programs that you chose to run on McKernel:

2.3.1. Non-MPI programs

Insert mcexec before the command:

mcexec <command>

2.3.2. MPI programs

Fujitsu MPI

Insert mcexec after mpirun and before an executable:

mpirun -n <number-of-MPI-processes> mcexec ./a.out

Other MPI

Insert mcexec -n <processes-per-node> after mpirun and before an executable:

mpirun -n <number-of-MPI-processes> mcexec -n <processes-per-node> ./a.out

<processes-per-node> is the number of the processes per node and calculated by (number of MPI processes) / (number of nodes).

For example, <processes-per-node> equals to 4 (=8/2) when specifying the number of processes and nodes as follows with MPICH.

mpirun -n 8 -hosts host1,host2 ./cpi

2.4. (Advanced) When using Utility Thread offloading Interface (UTI)

UTI enables a runtime such as MPI runtime to spawn utility threads such as MPI asynchronous progress threads to Linux cores.

Add --enable-uti option to mcexec:

mcexec --enable-uti <command>

3. Limitations

1. Pseudo devices such as /dev/mem and /dev/zero are not mmap()ed correctly even if the mmap() returns a success. An access of their mapping receives the SIGSEGV signal.

2. clone() supports only the following flags. All the other flags cause clone() to return error or are simply ignored.

   • CLONE_CHILD_CLEARTID

   • CLONE_CHILD_SETTID

   • CLONE_PARENT_SETTID

   • CLONE_SETTLS

   • CLONE_SIGHAND

   • CLONE_VM

3. PAPI has the following restriction.

   • Number of counters a user can use at the same time is up to the number of the physical counters in the processor.

4. msync writes back only the modified pages mapped by the calling process.

5. The following syscalls always return the ENOSYS error.

   • migrate_pages()

   • move_pages()

   • set_robust_list()

6. The following syscalls always return the EOPNOTSUPP error.

   • arch_prctl(ARCH_SET_GS)

   • signalfd()

7. signalfd4() returns a fd, but signal is not notified through the fd.

8. set_rlimit sets the limit values but they are not enforced.

9. Address randomization is not supported.

10. brk() extends the heap more than requestd when -h (–extend-heap-by=) option of mcexec is used with the value larger than 4 KiB. syscall_pwrite02 of LTP would fail for this reason. This is because the test expects that the end of the heap is set to the same address as the argument of sbrk() and expects a segmentation violation occurs when it tries to access the memory area right next to the boundary. However, the optimization sets the end to a value larger than the requested. Therefore, the expected segmentation violation doesn’t occur.

11. setpriority()/getpriority() won’t work. They might set/get the priority of a random mcexec thread. This is because there’s no fixed correspondence between a McKernel thread which issues the system call and a mcexec thread which handles the offload request.

12. mbind() can set the policy but it is not used when allocating physical pages.

13. MPOL_F_RELATIVE_NODES and MPOL_INTERLEAVE flags for set_mempolicy()/mbind() are not supported.

14. The MPOL_BIND policy for set_mempolicy()/mbind() works as the same as the MPOL_PREFERRED policy. That is, the physical page allocator doesn’t give up the allocation when the specified nodes are running out of pages but continues to search pages in the other nodes.

15. Kernel dump on Linux panic requires Linux kernel CentOS-7.4 and later. In addition, crash_kexec_post_notifiers kernel argument must be given to Linux kernel.

16. setfsuid()/setfsgid() cannot change the id of the calling thread. Instead, it changes that of the mcexec worker thread which takes the system-call offload request.

17. mmap (hugeTLBfs): The physical pages corresponding to a map are released when no McKernel process exist. The next map gets fresh physical pages.

18. Sticky bit on executable file has no effect.

19. Linux (RHEL-7 for x86_64) could hang when offlining CPUs in the process of booting McKernel due to the Linux bug, found in Linux-3.10 and fixed in the later version. One way to circumvent this is to always assign the same CPU set to McKernel.

20. madvise:

    • MADV_HWPOISON and MADV_SOFT_OFFLINE always returns -EPERM.

    • MADV_MERGEABLE and MADV_UNMERGEABLE always returns -EINVAL.

    • MADV_HUGEPAGE and MADV_NOHUGEPAGE on file map returns -EINVAL except on RHEL-8 for aarch64.

21. brk() and mmap() doesn’t report out-of-memory through its return value. Instead, page-fault reports the error.

22. Anonymous mmap pre-maps requested number of pages when contiguous pages are available. Demand paging is used when not available.

23. ihk_os_getperfevent() could time-out when invoked from Fujitsu TCS (job-scheduler).

24. The behaviors of madvise and mbind are changed to do nothing and report success as a workaround for Fugaku.

25. mmap() allows unlimited overcommit. Note that it corresponds to setting sysctl vm.overcommit_memory to 1.

26. mlockall() is not supported and returns -EPERM.

27. munlockall() is not supported and returns zero.

28. (Fujitsu TCS-only) A job following the one in which __mcctrl_os_read_write_cpu_register() returns -ETIME fails because xos_hwb related CPU state isn’t finalized. You can tell if the function returned -ETIME by checking if the following line appeared in the Linux kernel message:

    __mcctrl_os_read_write_cpu_register: ERROR sending IKC msg: -62
    

    You can re-initialize xos_hwb related CPU state by the following command:

    sudo systemctl restart xos_hwb
    

29. System calls can write the mcexec VMAs with PROT_WRITE flag not set. This is because we never turn off PROT_WRITE of the mcexec VMAs to circumvent the issue “set_host_vma(): do NOT read protect Linux VMA”.

30. procfs entry creation done by Linux work queue could starve when Linux CPUs are flooded with system call offloads. LTP-2019 sendmsg02 causes this issue.

This document will explain how to operate system with McKernel.

1. Installation

Follow Quick Guide – Installation.

2. Boot and Shut-down

2.1. Related files

In the followings, the install directory of IHK/McKernel is shown as <install> . The kernel modules and their locations are as follows.

<install>/kmod/ihk.ko	IHK-master core
<install>/kmod/ihk-smp-x86.ko	IHK-master driver
<install>/kmod/mcctrl.ko	Delegator module
<install>/smp-x86/kernel/mckernel.img	Kernel Image

The commands and daemons for operation and their locations are as follows.

<install>/sbin/mcreboot.sh	Boot script
<install>/sbin/mcstop+release.sh	Shutdown script
<install>/bin/mcexec	Process invocation command
<install>/bin/eclair	Kernel dump analysis tool
<install>/bin/vmcore2mckdump	Kernel dump format conversion tool

以下、関連コマンドおよび関連関数のインターフェイスを説明する。

2.2. インターフェイス

2.2.1. カーネル引数

McKernelのカーネル引数を表 Table 1 に示す。

McKernelのカーネル引数

hidos	IKCを有効にする。
dump_level=<level>	Linuxのpanicハンドラ経由でダンプを行った場合の、ダ ンプ対象とするメモリ領域の種類を<level>に設定する。 設定可能な値は以下の通り。 0 IHKがMcKernelに割り当てたメモリ領域を出力する 24 カーネルが使用しているメモリ領域を出力する 指定がなかった場合は24が用いられる。
allow_oversubscribe	McKernelに割り当てられたCPU数より大きい数のスレッド またはプロセスの生成を許可する。この引数が指定され ない場合に、CPU数より大きい数のスレッドまたはプロセ スをclone(), fork(), vfork()などで生成しようとする と、当該システムコールがEINVALエラーを返す。

2.2.2. ブートスクリプト

書式

mcreboot.sh -c <cpulist> -r <ikcmap> -m <memlist> -f <facility> -o <chownopt> -i <mon_interval> -k <redirct_kmsg> -q <irq> -t -d <dump_level> -O

オプション

-c <list>	McKernelに割り当てるCPUのリストを指定する。フォー マットは以下の通り。 <CPU logical id>,<CPU logical id>…または <CPU logical id>-<CPU logical id>,<CPU logical id> -<CPU logical id>…または両者の混合。
-r <map>	McKernelのCPUがIKCメッセージを送るLinux CPUを指定する。フォーマットは以下の通り。 <CPU list>:<CPU id>+<CPU list>:<CPU id>… <CPU list>のフォーマットは-cオプションにおけるもの と同じである。 各<CPU list>:<CPU id>は<CPU list>で示されるMcKernel のCPUが<CPU logical id>で示されるLinuxのCPUにIKC メッセージを送信することを意味する。
-m <list>	McKernelに割り当てるメモリ領域を指定する。フォーマッ トは以下の通り。 <size>@<NUMA-id>, <size>@<NUMA-id>…
-f <facility>	ihkmondが使用するsyslogプロトコルのfacilityを指定す る。デフォルトはLOG_LOCAL6。
-o <chownopt>	IHKのデバイスファイル(/dev/mcd*, /dev/mcos*)のオー ナーとグループの値を<user>[:<group>]の形式で指定す る。デフォルトはmcreboot.shを実行したユーザ。
-i <interval>	ihkmondがハングアップ検知のためにOS状態を確認する時 間間隔を秒単位で指定する。-1が指定された場合はハン グアップ検知を行わない。指定がない場合はハングアッ プ検知を行わない。
-k <redirect_kmsg>	カーネルメッセージの/dev/logへのリダイレクト有無を 指定する。0が指定された場合はリダイレクトを行わず、 0以外が指定された場合はリダイレクトを行う。指定がな い場合はリダイレクトを行わない。
-q <irq>	IHKが使用するIRQ番号を指定する。指定がない場合は 64-255の範囲で空いているものを使用する。
-t	（x86_64アーキテクチャのみ）Turbo Boostをオンにする。デフォルトはオフ。
-d <level>	Linuxのpanicハンドラ経由でダンプを行った場合の、ダ ンプ対象とするメモリ領域の種類を<level>に設定する。 設定可能な値は以下の通り。 0 IHKがMcKernelに割り当てたメモリ領域を出力する 24 カーネルが使用しているメモリ領域を出力する 指定がなかった場合は24が用いられる。
-O	McKernelに割り当てられたCPU数より大きい数のスレッド またはプロセスの生成を許可する。指定がない場合は許可 しない。すなわち、CPU数より大きい数のスレッドまたは プロセスを生成しようとするとエラーとなる。

説明

McKernel関連カーネルモジュールをinsmodし、<cpulist>で指定されたCPUと<memlist>で指定されたメモリ領域からなるパーティションを作成し、IKC mapを<ikcmap>に設定し、前記パーティションにMcKernelをブートする。

戻り値

0	正常終了
0以外	エラー

2.2.3. シャットダウンスクリプト

書式

mcstop+release.sh

オプション

なし

説明

McKernelをシャットダウンし、McKernel用パーティションを削除し、関連カーネルモジュールをrmmodする。

戻り値

0	正常終了
0以外	エラー

2.2.4. プロセス起動コマンド

2.2.5. ダンプ解析コマンド

2.2.6. ダンプ形式変換コマンド

2.3. ブート手順

mcreboot.shを用いてブート手順を説明する。

スクリプトは以下の通り。

#!/bin/bash

# IHK SMP-x86 example boot script.
# author: Balazs Gerofi <bgerofi@riken.jp>
#      Copyright (C) 2014  RIKEN AICS
#
# This is an example script for loading IHK, configuring a partition and
# booting McKernel on it.  Unless specific CPUs and memory are requested,
# the script reserves half of the CPU cores and 512MB of RAM from
# NUMA node 0 when IHK is loaded for the first time.
# Otherwise, it destroys the current McKernel instance and reboots it using
# the same set of resources as it used previously.
# Note that the script does not output anything unless an error occurs.

prefix="/home/takagi/project/os/install"
BINDIR="${prefix}/bin"
SBINDIR="${prefix}/sbin"
ETCDIR=/home/takagi/project/os/install/etc
KMODDIR="${prefix}/kmod"
KERNDIR="${prefix}/smp-x86/kernel"
ENABLE_MCOVERLAYFS="yes"

mem="512M@0"
cpus=""
ikc_map=""

if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then
     echo "You need at least bash-4.0 to run this script." >&2
     exit 1
fi

redirect_kmsg=0
mon_interval="-1"
DUMP_LEVEL=24
facility="LOG_LOCAL6"
chown_option=`logname 2> /dev/null`

if [ "`systemctl status irqbalance_mck.service 2> /dev/null |grep -E 'Active: active'`"\
 != "" -o "`systemctl status irqbalance.service 2> /dev/null |grep -E 'Active: active'`"\
 != "" ]; then
     irqbalance_used="yes"
else
     irqbalance_used="no"
fi

turbo=""
ihk_irq=""

while getopts :tk:c:m:o:f:r:q:i:d: OPT
do
     case ${OPT} in
     f)      facility=${OPTARG}
             ;;
     o)      chown_option=${OPTARG}
             ;;
     k)      redirect_kmsg=${OPTARG}
             ;;
     c) cpus=${OPTARG}
             ;;
     m) mem=${OPTARG}
             ;;
     r) ikc_map=${OPTARG}
             ;;
     q) ihk_irq=${OPTARG}
             ;;
     t) turbo="turbo"
             ;;
     d) DUMP_LEVEL=${OPTARG}
             ;;
     i) mon_interval=${OPTARG}
             ;;
     *)  echo "invalid option -${OPT}" >&2
             exit 1
     esac
done

# Start ihkmond
pid=`pidof ihkmond`
if [ "${pid}" != "" ]; then
    sudo kill -9 ${pid} > /dev/null 2> /dev/null
fi
if [ "${redirect_kmsg}" != "0" -o "${mon_interval}" != "-1" ]; then
    ${SBINDIR}/ihkmond -f ${facility} -k ${redirect_kmsg} -i ${mon_interval}
fi
#
# Revert any state that has been initialized before the error occured.
#
error_exit() {
     local status=$1

     case $status in
     mcos_sys_mounted)
             if [ "$enable_mcoverlay" == "yes" ]; then
                     umount /tmp/mcos/mcos0_sys
             fi
             ;&
     mcos_proc_mounted)
             if [ "$enable_mcoverlay" == "yes" ]; then
                     umount /tmp/mcos/mcos0_proc
             fi
             ;&
     mcoverlayfs_loaded)
             if [ "$enable_mcoverlay" == "yes" ]; then
                     rmmod mcoverlay 2>/dev/null
             fi
             ;&
     linux_proc_bind_mounted)
             if [ "$enable_mcoverlay" == "yes" ]; then
                     umount /tmp/mcos/linux_proc
             fi
             ;&
     tmp_mcos_mounted)
             if [ "$enable_mcoverlay" == "yes" ]; then
                     umount /tmp/mcos
             fi
             ;&
     tmp_mcos_created)
             if [ "$enable_mcoverlay" == "yes" ]; then
                     rm -rf /tmp/mcos
             fi
             ;&
     os_created)
             # Destroy all LWK instances
             if ls /dev/mcos* 1>/dev/null 2>&1; then
                     for i in /dev/mcos*; do
                             ind=`echo $i|cut -c10-`;
                             if ! ${SBINDIR}/ihkconfig 0 destroy $ind; then
                                     echo "warning: failed to destroy LWK instance $ind" >&2
                             fi
                     done
             fi
             ;&
     mcctrl_loaded)
             rmmod mcctrl 2>/dev/null || echo "warning: failed to remove mcctrl" >&2
             ;&
     cpus_reserved)
             cpus=`${SBINDIR}/ihkconfig 0 query cpu`
             if [ "${cpus}" != "" ]; then
                     if ! ${SBINDIR}/ihkconfig 0 release cpu $cpus > /dev/null; then
                             echo "warning: failed to release CPUs" >&2
                     fi
             fi
             ;&
     mem_reserved)
             mem=`${SBINDIR}/ihkconfig 0 query mem`
             if [ "${mem}" != "" ]; then
                     if ! ${SBINDIR}/ihkconfig 0 release mem $mem > /dev/null; then
                             echo "warning: failed to release memory" >&2
                     fi
             fi
             ;&
     ihk_smp_loaded)
             rmmod ihk_smp_x86 2>/dev/null || echo "warning: failed to remove ihk_smp_x86" >&2
             ;&
     ihk_loaded)
             rmmod ihk 2>/dev/null || echo "warning: failed to remove ihk" >&2
             ;&
     irqbalance_stopped)
             if [ "`systemctl status irqbalance_mck.service 2> /dev/null |'\
'grep -E 'Active: active'`" != "" ]; then
                     if ! systemctl stop irqbalance_mck.service 2>/dev/null; then
                             echo "warning: failed to stop irqbalance_mck" >&2
                     fi
                     if ! systemctl disable irqbalance_mck.service >/dev/null 2>/dev/null; then
                             echo "warning: failed to disable irqbalance_mck" >&2
                     fi
                     if ! etcdir=/home/takagi/project/os/install/etc perl -e \
'$etcdir=$ENV{'etcdir'}; @files = grep { -f } glob "$etcdir/proc/irq/*/smp_affinity";'\
' foreach $file (@files) { $dest = substr($file, length($etcdir));'\
' if(0) {print "cp $file $dest\n";} system("cp $file $dest 2>/dev/null"); }'; then
                             echo "warning: failed to restore /proc/irq/*/smp_affinity" >&2
                     fi
                     if ! systemctl start irqbalance.service; then
                             echo "warning: failed to start irqbalance" >&2;
                     fi
             fi
             ;&
     initial)
             # Nothing more to revert
             ;;
     esac

     exit 1
}

ihk_ikc_irq_core=0

release=`uname -r`
major=`echo ${release} | sed -e 's/^\([0-9]*\).*/\1/'`
minor=`echo ${release} | sed -e 's/^[0-9]*.\([0-9]*\).*/\1/'`
patch=`echo ${release} | sed -e 's/^[0-9]*.[0-9]*.\([0-9]*\).*/\1/'`
linux_version_code=`expr \( ${major} \* 65536 \) + \( ${minor} \* 256 \) + ${patch}`
rhel_release=`echo ${release} | sed -e 's/^[0-9]*.[0-9]*.[0-9]*-\([0-9]*\).*/\1/'`
if [ "${release}" == "${rhel_release}" ]; then
     rhel_release="";
fi

enable_mcoverlay="no"

if [ "${ENABLE_MCOVERLAYFS}" == "yes" ]; then
     if [ "${rhel_release}" == "" ]; then
             if [ ${linux_version_code} -ge 262144 -a ${linux_version_code} -lt 262400 ]; then
                     enable_mcoverlay="yes"
             fi
             if [ ${linux_version_code} -ge 263680 -a ${linux_version_code} -lt 263936 ]; then
                     enable_mcoverlay="yes"
             fi
     else
             if [ ${linux_version_code} -eq 199168 -a ${rhel_release} -ge 327 -a ${rhel_release} -le 693 ]; then
                     enable_mcoverlay="yes"
             fi
             if [ ${linux_version_code} -ge 262144 -a ${linux_version_code} -lt 262400 ]; then
                     enable_mcoverlay="yes"
             fi
     fi
fi

# Figure out CPUs if not requested by user
if [ "$cpus" == "" ]; then
     # Get the number of CPUs on NUMA node 0
     nr_cpus=`lscpu --parse | awk -F"," '{if ($4 == 0) print $4}' | wc -l`

     # Use the second half of the cores
     let nr_cpus="$nr_cpus / 2"
     cpus=`lscpu --parse | awk -F"," '{if ($4 == 0) print $1}' | tail -n $nr_cpus |'\
' xargs echo -n | sed 's/ /,/g'`
     if [ "$cpus" == "" ]; then
             echo "error: no available CPUs on NUMA node 0?" >&2
             exit 1
     fi
fi

# Remove mcoverlay if loaded
if [ "$enable_mcoverlay" == "yes" ]; then
     if grep mcoverlay /proc/modules &>/dev/null; then
             if [ "`cat /proc/mounts | grep /tmp/mcos/mcos0_sys`" != "" ]; \
then umount -l /tmp/mcos/mcos0_sys; fi
             if [ "`cat /proc/mounts | grep /tmp/mcos/mcos0_proc`" != "" ]; \
then umount -l /tmp/mcos/mcos0_proc; fi
             if [ "`cat /proc/mounts | grep /tmp/mcos/linux_proc`" != "" ]; \
then umount -l /tmp/mcos/linux_proc; fi
             if [ "`cat /proc/mounts | grep /tmp/mcos`" != "" ]; then umount -l /tmp/mcos; fi
             if [ -e /tmp/mcos ]; then rm -rf /tmp/mcos; fi
             if ! rmmod mcoverlay 2>/dev/null; then
                     echo "error: removing mcoverlay" >&2
                     exit 1
             fi
     fi
fi

# Stop irqbalance
if [ "${irqbalance_used}" == "yes" ]; then
    systemctl stop irqbalance_mck.service 2>/dev/null
    if ! systemctl stop irqbalance.service 2>/dev/null ; then
             echo "error: stopping irqbalance" >&2
             exit 1
    fi;

    if ! etcdir=/home/takagi/project/os/install/etc perl -e \
'use File::Copy qw(copy); $etcdir=$ENV{'etcdir'}; '\
'@files = grep { -f } glob "/proc/irq/*/smp_affinity"; foreach $file (@files) { '\
'$rel = substr($file, 1); $dir=substr($rel, 0, length($rel)-length("/smp_affinity")); '\
'if(0) { print "cp $file $etcdir/$rel\n";} if(system("mkdir -p $etcdir/$dir")){ exit 1;} '\
'if(!copy($file,"$etcdir/$rel")){ exit 1;} }'; then
             echo "error: saving /proc/irq/*/smp_affinity" >&2
             error_exit "mcos_sys_mounted"
     fi;

# Prevent /proc/irq/*/smp_affinity from getting zero after offlining
# McKernel CPUs by using the following algorithm.
# if (smp_affinity & mck_cores) {
#     smp_affinity = (mck_cores ^ -1);
# }
    ncpus=`lscpu | grep -E '^CPU\(s\):' | awk '{print $2}'`
    smp_affinity_mask=`echo $cpus | ncpus=$ncpus perl -e \
'while(<>){@tokens = split /,/;foreach $token (@tokens) '\
'{@nums = split /-/,$token; for($num = $nums[0]; $num <= $nums[$#nums]; $num++) {'\
'$ndx=int($num/32); $mask[$ndx] |= (1<<($num % 32))}}}'\
' $nint32s = int(($ENV{'ncpus'}+31)/32); for($j = $nint32s - 1; $j >= 0; $j--) {'\
' if($j != $nint32s - 1){print ",";}'\
' $nblks = ($j != $nint32s - 1) ? 8 : ($ENV{'ncpus'} % 32 != 0) ? '\
'int((($ENV{'ncpus'} + 3) % 32) / 4) : 8;'\
' for($i = $nblks - 1;$i >= 0;$i--){ printf("%01x",($mask[$j] >> ($i*4)) & 0xf);}}'`
#    echo cpus=$cpus ncpus=$ncpus smp_affinity_mask=$smp_affinity_mask

    if ! ncpus=$ncpus smp_affinity_mask=$smp_affinity_mask perl -e \
'@dirs = grep { -d } glob "/proc/irq/*"; foreach $dir (@dirs) {'\
' $hit = 0; $affinity_str = `cat $dir/smp_affinity`; chomp $affinity_str;'\
' @int32strs = split /,/, $affinity_str; @int32strs_mask=split /,/, $ENV{'smp_affinity_mask'};'\
' for($i=0;$i <= $#int32strs_mask; $i++) {'\
' $int32strs_inv[$i] = sprintf("%08x",hex($int32strs_mask[$i])^0xffffffff);'\
' if($i == 0) { $len = int((($ENV{'ncpus'}%32)+3)/4); if($len != 0) {'\
' $int32strs_inv[$i] = substr($int32strs_inv[$i], -$len, $len); } } }'\
' $inv = join(",", @int32strs_inv); $nint32s = int(($ENV{'ncpus'}+31)/32);'\
' for($j = $nint32s - 1; $j >= 0; $j--) {'\
' if(hex($int32strs[$nint32s - 1 - $j]) & hex($int32strs_mask[$nint32s - 1 - $j])) {'\
' $hit = 1; }} if($hit == 1) {'\
' $cmd = "echo $inv > $dir/smp_affinity 2>/dev/null"; system $cmd;}}'; then
             echo "error: modifying /proc/irq/*/smp_affinity" >&2
             error_exit "mcos_sys_mounted"
     fi

fi

# Load IHK if not loaded
if ! grep -E 'ihk\s' /proc/modules &>/dev/null; then
     if ! taskset -c 0 insmod ${KMODDIR}/ihk.ko 2>/dev/null; then
             echo "error: loading ihk" >&2
             error_exit "irqbalance_stopped"
     fi
fi

# Increase swappiness so that we have better chance to allocate memory for IHK
echo 100 > /proc/sys/vm/swappiness

# Drop Linux caches to free memory
sync && echo 3 > /proc/sys/vm/drop_caches

# Merge free memory areas into large, physically contigous ones
echo 1 > /proc/sys/vm/compact_memory 2>/dev/null

sync

# Load IHK-SMP if not loaded and reserve CPUs and memory
if ! grep ihk_smp_x86 /proc/modules &>/dev/null; then
     if [ "$ihk_irq" == "" ]; then
             for i in `seq 64 255`; do
                     if [ ! -d /proc/irq/$i ] && \
[ "`cat /proc/interrupts | grep ":" | awk '{print $1}' | grep -o '[0-9]*' | grep -e '^$i$'`"\
 == "" ]; then
                             ihk_irq=$i
                             break
                     fi
             done
             if [ "$ihk_irq" == "" ]; then
                     echo "error: no IRQ available" >&2
                     error_exit "ihk_loaded"
             fi
     fi
     if ! taskset -c 0 insmod ${KMODDIR}/ihk-smp-x86.ko ihk_start_irq=$ihk_irq\
 ihk_ikc_irq_core=$ihk_ikc_irq_core 2>/dev/null; then
             echo "error: loading ihk-smp-x86" >&2
             error_exit "ihk_loaded"
     fi

     # Offline-reonline RAM (special case for OFP SNC-4 mode)
     if [ "`hostname | grep "c[0-9][0-9][0-9][0-9].ofp"`" != "" ] && [ "`cat /sys/devices/system/node/online`" == "0-7" ]; then
             for i in  0 1 2 3; do
                     find /sys/devices/system/node/node$i/memory*/ -name "online" |\
 while read f; do
                             echo 0 > $f 2>&1 > /dev/null;
                     done
                     find /sys/devices/system/node/node$i/memory*/ -name "online" |\
 while read f; do
                             echo 1 > $f 2>&1 > /dev/null;
                     done
             done
             for i in 4 5 6 7; do
                     find /sys/devices/system/node/node$i/memory*/ -name "online" |\
 while read f; do
                             echo 0 > $f 2>&1 > /dev/null;
                     done
                     find /sys/devices/system/node/node$i/memory*/ -name "online" |\
 while read f; do
                             echo 1 > $f 2>&1 > /dev/null;
                     done
             done
     fi

     if ! ${SBINDIR}/ihkconfig 0 reserve mem ${mem}; then
             echo "error: reserving memory" >&2
             error_exit "ihk_smp_loaded"
     fi
     if ! ${SBINDIR}/ihkconfig 0 reserve cpu ${cpus}; then
             echo "error: reserving CPUs" >&2;
             error_exit "mem_reserved"
     fi
fi

# Load mcctrl if not loaded
if ! grep mcctrl /proc/modules &>/dev/null; then
     if ! taskset -c 0 insmod ${KMODDIR}/mcctrl.ko 2>/dev/null; then
             echo "error: inserting mcctrl.ko" >&2
             error_exit "cpus_reserved"
     fi
fi

# Destroy all LWK instances
if ls /dev/mcos* 1>/dev/null 2>&1; then
     for i in /dev/mcos*; do
             ind=`echo $i|cut -c10-`;
             # Retry when conflicting with ihkmond
             nretry=0
             until ${SBINDIR}/ihkconfig 0 destroy $ind || [ $nretry -lt 4 ]; do
                 sleep 0.25
                 nretry=$[ $nretry + 1 ]
             done
             if [ $nretry -eq 4 ]; then
                 echo "error: destroying LWK instance $ind failed" >&2
                 error_exit "mcctrl_loaded"
             fi
     done
fi

# Create OS instance
if ! ${SBINDIR}/ihkconfig 0 create; then
     echo "error: creating OS instance" >&2
     error_exit "mcctrl_loaded"
fi

# Assign CPUs
if ! ${SBINDIR}/ihkosctl 0 assign cpu ${cpus}; then
     echo "error: assign CPUs" >&2
     error_exit "os_created"
fi

if [ "$ikc_map" != "" ]; then
     # Specify IKC map
     if ! ${SBINDIR}/ihkosctl 0 set ikc_map ${ikc_map}; then
             echo "error: assign CPUs" >&2
             error_exit "os_created"
     fi
fi

# Assign memory
if ! ${SBINDIR}/ihkosctl 0 assign mem ${mem}; then
     echo "error: assign memory" >&2
     error_exit "os_created"
fi

# Load kernel image
if ! ${SBINDIR}/ihkosctl 0 load ${KERNDIR}/mckernel.img; then
     echo "error: loading kernel image: ${KERNDIR}/mckernel.img" >&2
     error_exit "os_created"
fi

# Set kernel arguments
if ! ${SBINDIR}/ihkosctl 0 kargs "hidos $turbo dump_level=${DUMP_LEVEL}"; then
     echo "error: setting kernel arguments" >&2
     error_exit "os_created"
fi

# Boot OS instance
if ! ${SBINDIR}/ihkosctl 0 boot; then
     echo "error: booting" >&2
     error_exit "os_created"
fi

# Set device file ownership
if ! chown ${chown_option} /dev/mcd* /dev/mcos*; then
     echo "warning: failed to chown device files" >&2
fi

# Overlay /proc, /sys with McKernel specific contents
if [ "$enable_mcoverlay" == "yes" ]; then
     if [ ! -e /tmp/mcos ]; then
             mkdir -p /tmp/mcos;
     fi
     if ! mount -t tmpfs tmpfs /tmp/mcos; then
             echo "error: mount /tmp/mcos" >&2
             error_exit "tmp_mcos_created"
     fi
     if [ ! -e /tmp/mcos/linux_proc ]; then
             mkdir -p /tmp/mcos/linux_proc;
     fi
     if ! mount --bind /proc /tmp/mcos/linux_proc; then
             echo "error: mount /tmp/mcos/linux_proc" >&2
             error_exit "tmp_mcos_mounted"
     fi
     if ! taskset -c 0 insmod ${KMODDIR}/mcoverlay.ko 2>/dev/null; then
             echo "error: inserting mcoverlay.ko" >&2
             error_exit "linux_proc_bind_mounted"
     fi
     while [ ! -e /proc/mcos0 ]
     do
             sleep 0.1
     done
     if [ ! -e /tmp/mcos/mcos0_proc ]; then
             mkdir -p /tmp/mcos/mcos0_proc;
     fi
     if [ ! -e /tmp/mcos/mcos0_proc_upper ]; then
             mkdir -p /tmp/mcos/mcos0_proc_upper;
     fi
     if [ ! -e /tmp/mcos/mcos0_proc_work ]; then
             mkdir -p /tmp/mcos/mcos0_proc_work;
     fi
     if ! mount -t mcoverlay mcoverlay -o\
 lowerdir=/proc/mcos0:/proc,upperdir=/tmp/mcos/mcos0_proc_upper,\
workdir=/tmp/mcos/mcos0_proc_work,nocopyupw,nofscheck /tmp/mcos/mcos0_proc; then
             echo "error: mounting /tmp/mcos/mcos0_proc" >&2
             error_exit "mcoverlayfs_loaded"
     fi
     # TODO: How de we revert this in case of failure??
     mount --make-rprivate /proc

     while [ ! -e /sys/devices/virtual/mcos/mcos0/sys/setup_complete ]
     do
             sleep 0.1
     done
     if [ ! -e /tmp/mcos/mcos0_sys ]; then
             mkdir -p /tmp/mcos/mcos0_sys;
     fi
     if [ ! -e /tmp/mcos/mcos0_sys_upper ]; then
             mkdir -p /tmp/mcos/mcos0_sys_upper;
     fi
     if [ ! -e /tmp/mcos/mcos0_sys_work ]; then
             mkdir -p /tmp/mcos/mcos0_sys_work;
     fi
     if ! mount -t mcoverlay mcoverlay -o\
 lowerdir=/sys/devices/virtual/mcos/mcos0/sys:/sys,upperdir=/tmp/mcos/mcos0_sys_upper,\
workdir=/tmp/mcos/mcos0_sys_work,nocopyupw,nofscheck /tmp/mcos/mcos0_sys; then
             echo "error: mount /tmp/mcos/mcos0_sys" >&2
             error_exit "mcos_proc_mounted"
     fi
     # TODO: How de we revert this in case of failure??
     mount --make-rprivate /sys

     touch /tmp/mcos/mcos0_proc/mckernel

     rm -rf /tmp/mcos/mcos0_sys/setup_complete

     # Hide NUMA related files which are outside the LWK partition
     for cpuid in \
`find /sys/devices/system/cpu/* -maxdepth 0 -name "cpu[0123456789]*" -printf "%f "`; do
             if [ ! -e "/sys/devices/virtual/mcos/mcos0/sys/devices/system/cpu/$cpuid" ]; then
                     rm -rf /tmp/mcos/mcos0_sys/devices/system/cpu/$cpuid
                     rm -rf /tmp/mcos/mcos0_sys/bus/cpu/devices/$cpuid
                     rm -rf /tmp/mcos/mcos0_sys/bus/cpu/drivers/processor/$cpuid
             else
                     for nodeid in \
`find /sys/devices/system/cpu/$cpuid/* -maxdepth 0 -name "node[0123456789]*" -printf "%f "`; do
                             if [ ! -e \
"/sys/devices/virtual/mcos/mcos0/sys/devices/system/cpu/$cpuid/$nodeid" ]; then
                                     rm -f \
/tmp/mcos/mcos0_sys/devices/system/cpu/$cpuid/$nodeid
                             fi
                     done
             fi
     done
     for nodeid in \
`find /sys/devices/system/node/* -maxdepth 0 -name "node[0123456789]*" -printf "%f "`; do
             if [ ! -e "/sys/devices/virtual/mcos/mcos0/sys/devices/system/node/$nodeid" ]; \
then
                     rm -rf /tmp/mcos/mcos0_sys/devices/system/node/$nodeid/*
                     rm -rf /tmp/mcos/mcos0_sys/bus/node/devices/$nodeid
             else
                     # Delete non-existent symlinks
                     for cpuid in \
`find /sys/devices/system/node/$nodeid/* -maxdepth 0 -name "cpu[0123456789]*" -printf "%f "`; do
                             if [ ! -e \
"/sys/devices/virtual/mcos/mcos0/sys/devices/system/node/$nodeid/$cpuid" ]; then
                                     rm -f \
/tmp/mcos/mcos0_sys/devices/system/node/$nodeid/$cpuid
                             fi
                     done

                     rm -f /tmp/mcos/mcos0_sys/devices/system/node/$nodeid/memory*
             fi
     done
     rm -f /tmp/mcos/mcos0_sys/devices/system/node/has_*
     for cpuid in \
`find /sys/bus/cpu/devices/* -maxdepth 0 -name "cpu[0123456789]*" -printf "%f "`; do
             if [ ! -e "/sys/devices/virtual/mcos/mcos0/sys/bus/cpu/devices/$cpuid" ]; then
                     rm -rf /tmp/mcos/mcos0_sys/bus/cpu/devices/$cpuid
             fi
     done
fi

# Start irqbalance with CPUs and IRQ for McKernel banned
if [ "${irqbalance_used}" == "yes" ]; then
    banirq=`cat /proc/interrupts| \
perl -e 'while(<>) { if(/^\s*(\d+).*IHK\-SMP\s*$/) {print $1;}}'`

    sed "s/%mask%/$smp_affinity_mask/g" $ETCDIR/irqbalance_mck.in | \
sed "s/%banirq%/$banirq/g" > /tmp/irqbalance_mck
     systemctl disable irqbalance_mck.service >/dev/null 2>/dev/null
     if ! systemctl link $ETCDIR/irqbalance_mck.service >/dev/null 2>/dev/null; then
             echo "error: linking irqbalance_mck" >&2
             error_exit "mcos_sys_mounted"
     fi

    if ! systemctl start irqbalance_mck.service 2>/dev/null ; then
             echo "error: starting irqbalance_mck" >&2
             error_exit "mcos_sys_mounted"
     fi
#    echo cpus=$cpus ncpus=$ncpus banirq=$banirq
fi

手順は以下の通り。

1. ihkmondを起動する。ihkmondは任意のタイミングで起動してよい。これは、ihkmondはOSインスタンスの作成を検知して動作を開始するためである。（83行目）

2. Linuxのカーネルバージョンが、mcoverlayfsが動作するものであるかを確認する。（200–216行目）

3. irqbalanceを停止する。（251–257行目）

4. /proc/irq/[n]/affinityの設定を保存した上でMcKernel CPUを担当から外す。担当CPUが無くなる場合は、全てのLinux CPUを指定する。（269–303行目）

5. ihk.koをinsmodする。（307行目）

6. Linuxによるメモリフラグメンテーションを緩和するために以下を実施する。（313–320行目）

   1. アクティブでないプロセスを積極的にスワップアウトするように設定する

   2. クリーンなページキャッシュを無効化し、またdentriesやinodeのslabオブジェクトのうち可能なものを破棄する

   3. 連続する空き領域を結合してより大きな空き領域にまとめる

7. ihk-smp-x86.koをinsmodする。（340行目）ihk-smp-x86.koは関数をihk.koに登録する。このため、ihk-smp-x86.koはihk.koをinsmodした後にinsmodする必要がある。

8. メモリを予約する。（370行目）

9. CPUを予約する。（374行目）

10. McKernelのカーネルモジュールmcctrl.koをinsmodする。（382行目）mcctrl.koはMcKernelブート時に呼ばれる関数をihk.koに登録する。このため、mcctrl.koのinsmodはihk.koのinsmodの後に、またブートの前に行う必要がある。

11. OSインスタンスを作成する。（406行目）

12. OSインスタンスにCPUを割り当てる。（412行目）

13. McKernel CPUのIKCメッセージ送信先のLinux CPUを設定する。（419行目）

14. OSインスタンスにメモリを割り当てる。（426行目）

15. カーネルイメージをロードする。（432行目）

16. カーネル引数をカーネルに渡す。（438行目）

17. カーネルをブートする。（444行目）

18. /proc, /sysファイルの準備をする。また、その中でmcoverlayfs.koをinsmodする。mcoverlayfs.koは他モジュールとの依存関係を持たない。（454行目から567行目）なお、関数インターフェイスでの対応関数はihk_os_create_pseudofs()である。

19. irqbalanceを、Linux CPUのみを対象とする設定で開始する。（569–587行目）

2.4. シャットダウン手順

mcstop+release.shを用いてシャットダウン手順を説明する。

スクリプトは以下の通り。

#!/bin/bash

# IHK SMP-x86 example McKernel unload script.
# author: Balazs Gerofi <bgerofi@riken.jp>
#      Copyright (C) 2015  RIKEN AICS
#
# This is an example script for destroying McKernel and releasing IHK resources
# Note that the script does no output anything unless an error occurs.

prefix="/home/takagi/project/os/install"
BINDIR="/home/takagi/project/os/install/bin"
SBINDIR="/home/takagi/project/os/install/sbin"
ETCDIR=/home/takagi/project/os/install/etc
KMODDIR="/home/takagi/project/os/install/kmod"
KERNDIR="/home/takagi/project/os/install/smp-x86/kernel"

mem=""
cpus=""
irqbalance_used=""

# No SMP module? Exit.
if ! grep ihk_smp_x86 /proc/modules &>/dev/null; then exit 0; fi

if [ "`systemctl status irqbalance_mck.service 2> /dev/null |grep -E 'Active: active'`" \
!= "" ]; then
     irqbalance_used="yes"
     if ! systemctl stop irqbalance_mck.service 2>/dev/null; then
             echo "warning: failed to stop irqbalance_mck" >&2
     fi
     if ! systemctl disable irqbalance_mck.service >/dev/null 2>/dev/null; then
             echo "warning: failed to disable irqbalance_mck" >&2
     fi
fi

# Destroy all LWK instances
if ls /dev/mcos* 1>/dev/null 2>&1; then
     for i in /dev/mcos*; do
             ind=`echo $i|cut -c10-`;
             # Retry when conflicting with ihkmond
             nretry=0
             until ${SBINDIR}/ihkconfig 0 destroy $ind || [ $nretry -lt 4 ]; do
                 sleep 0.25
                 nretry=$[ $nretry + 1 ]
             done
             if [ $nretry -eq 4 ]; then
                 echo "error: destroying LWK instance $ind failed" >&2
                 exit 1
             fi
     done
fi

# Query IHK-SMP resources and release them
if ! ${SBINDIR}/ihkconfig 0 query cpu > /dev/null; then
     echo "error: querying cpus" >&2
     exit 1
fi

cpus=`${SBINDIR}/ihkconfig 0 query cpu`
if [ "${cpus}" != "" ]; then
     if ! ${SBINDIR}/ihkconfig 0 release cpu $cpus > /dev/null; then
             echo "error: releasing CPUs" >&2
             exit 1
     fi
fi

if ! ${SBINDIR}/ihkconfig 0 query mem > /dev/null; then
     echo "error: querying memory" >&2
     exit 1
fi

mem=`${SBINDIR}/ihkconfig 0 query mem`
if [ "${mem}" != "" ]; then
     if ! ${SBINDIR}/ihkconfig 0 release mem $mem > /dev/null; then
             echo "error: releasing memory" >&2
             exit 1
     fi
fi

# Remove delegator if loaded
if grep mcctrl /proc/modules &>/dev/null; then
     if ! rmmod mcctrl 2>/dev/null; then
             echo "error: removing mcctrl" >&2
             exit 1
     fi
fi

# Remove mcoverlay if loaded
if grep mcoverlay /proc/modules &>/dev/null; then
     if [ "`cat /proc/mounts | grep /tmp/mcos/mcos0_sys`" != "" ]; \
then umount -l /tmp/mcos/mcos0_sys; fi
     if [ "`cat /proc/mounts | grep /tmp/mcos/mcos0_proc`" != "" ]; \
then umount -l /tmp/mcos/mcos0_proc; fi
     if [ "`cat /proc/mounts | grep /tmp/mcos/linux_proc`" != "" ]; \
then umount -l /tmp/mcos/linux_proc; fi
     if [ "`cat /proc/mounts | grep /tmp/mcos`" != "" ]; then umount -l /tmp/mcos; fi
     if [ -e /tmp/mcos ]; then rm -rf /tmp/mcos; fi
     if ! rmmod mcoverlay 2>/dev/null; then
             echo "warning: failed to remove mcoverlay" >&2
     fi
fi

# Remove SMP module
if grep ihk_smp_x86 /proc/modules &>/dev/null; then
     if ! rmmod ihk_smp_x86 2>/dev/null; then
             echo "error: removing ihk_smp_x86" >&2
             exit 1
     fi
fi

# Remove core module
if grep -E 'ihk\s' /proc/modules &>/dev/null; then
     if ! rmmod ihk 2>/dev/null; then
             echo "error: removing ihk" >&2
             exit 1
     fi
fi

# Stop ihkmond
pid=`pidof ihkmond`
if [ "${pid}" != "" ]; then
    sudo kill -9 ${pid} > /dev/null 2> /dev/null
fi

# Start irqbalance with the original settings
if [ "${irqbalance_used}" != "" ]; then
     if ! etcdir=/home/takagi/project/os/install/etc perl -e \
'$etcdir=$ENV{'etcdir'}; @files = grep { -f } glob "$etcdir/proc/irq/*/smp_affinity";'\
' foreach $file (@files) { $dest = substr($file, length($etcdir));'\
' if(0) {print "cp $file $dest\n";} system("cp $file $dest 2>/dev/null"); }'; then
             echo "warning: failed to restore /proc/irq/*/smp_affinity" >&2
     fi
     if ! systemctl start irqbalance.service; then
             echo "warning: failed to start irqbalance" >&2;
     fi
fi

# Set back default swappiness
echo 60 > /proc/sys/vm/swappiness

手順は以下の通り。

1. ブート時にLinux CPUのみを対象とする設定で開始されたirqbalanceを停止する。（24–33行目）

2. 全てのOSインスタンスを破壊する。OSインスタンスに割り当てられていた資源はIHKがLWKのために予約した状態に移行する。（35–50行目）

3. IHKがLWKのために予約していた資源を開放する。（52–77行目）

4. mcctrl.koをrmmodする。（81行目）

5. /proc, /sysファイルの準備をする。また、その中でmcoverlayfs.koをrmmodする。（87–100行目）なお、関数インターフェイスでの対応関数はihk_os_destroy_pseudofs()である。

6. ihk-smp-x86.koをrmmodする。（104行目）

7. ihk.koをrmmodする。（112行目）

8. ihkmondを停止する。（121行目）

9. /proc/irq/[n]/affinityの設定をブート時に保存しておいたものに戻し、ブート前の設定でirqbalanceを開始する。（124–135行目）

10. Linuxカーネルのスワップアウト積極度の設定をデフォルトの値に戻す。（138行目）

1. External Specs

1.1. Overview

1.2. Function Specs

1.3. Command / Daemon Specs

2. Booting LWK

1. Interfaces

2. Interface details

1. Specifications

The specifications pdf is here

1. Specifications

The specifications pdf is here

Version 1.7.9 (Mar 17, 2021)

IHK major updates

N/A

IHK major bug fixes

1. ihklib: ihk_reserve_mem_conf*: fix default values

2. smp_ihk_os_shutdown: fix memory leak

3. smp_ihk_os_shutdown: prevent double free

4. __ihk_os_shutdown: fix smp_ihk_os_shutdown()-related double free

5. smp_ihk_os_panic_notifier: exclude memory from Linux dump with default setting

6. smp_ihk_os_panic_notifier: exclude memory from Linux dump while booting, on timeout

McKernel major updates

N/A

McKernel major bug fixes

1. mcctrl_wakeup_desc: refcount and fix timeouts

Version 1.7.8 (Mar 12, 2021)

IHK major updates

N/A

IHK major bug fixes

1. ihklib: ihk_reserve_cpu: fix job cpu check when using krm

McKernel major updates

N/A

McKernel major bug fixes

N/A

Version 1.7.7 (Mar 11, 2021)

IHK major updates

N/A

IHK major bug fixes

N/A

McKernel major updates

N/A

McKernel major bug fixes

1. mcexec: fput executable just after its contents is transferred

2. spec: cmake-config cmake parameters

Version 1.7.6 (Mar 11, 2021)

IHK major updates

N/A

IHK major bug fixes

1. ihklib: ihk_reserve_mem_conf*: apply change only to the next reservation

McKernel major updates

N/A

McKernel major bug fixes

N/A

Version 1.7.5 (Mar 11, 2021)

IHK major updates

N/A

IHK major bug fixes

1. ihklib: fix cgroup cpuset.cpus/mems check when using krm

2. ihklib: ihk_reserve_mem_conf_str: set default values to those not specified

McKernel major updates

N/A

McKernel major bug fixes

N/A

Version 1.7.4 (Mar 7, 2021)

IHK major updates

N/A

IHK major bug fixes

N/A

McKernel major updates

N/A

McKernel major bug fixes

N/A

Version 1.7.3 (Mar 5, 2021)

IHK major updates

N/A

IHK major bug fixes

N/A

McKernel major updates

N/A

McKernel major bug fixes

N/A

Version 1.7.2 (Mar 5, 2021)

IHK major updates

1. ihklib: add *_str() functions for reserve, assign, IKC-map, kargs

2. smp: make smp_call_func() arch independent

IHK major bug fixes

1. ihklib: ihk_reserve_mem: fix capped best-effort

2. TO RESET: fake missing NUMA node pieces, 90% memory limit

3. ihklib: ihk_reserve_mem_conf: range-check for IHK_RESERVE_MEM_MAX_SIZE_RATIO_ALL

4. ihklib: ihk_os_kargs: check if “hidos” is included

5. SMP: omit slab/slub shrink, use 95% limit by default

6. check cpu / numa cgroup set by krm

7. SMP: __ihk_smp_reserve_mem: add __GFP_COMP to __GFP_ATOMIC allocation

8. ihk_register_device: record minor to IHK device object

McKernel major updates

1. mcexec: memory policy control by environmental variable

2. mempolicy: Support MPOL_INTERLEAVE

3. uti: futex call function in mcctrl

4. uti: integrate libuti and redirect to mck/libuti.so

5. uti: integrate syscall_intercept

6. shmobj: support large page

7. xpmem: support large page

8. MM: handle zero_at_free in page faults

McKernel major bug fixes

1. TO RESET: stack changes

2. Tofu: keep track of stags per memory range

3. Tofu: match page sizes to MBPT and fault PTEs if not present

4. Tofu: fix phys addr calculation for contiguous pages in MBPT/BCH update

5. rus_vm_fault: vmf_insert_pfn: treat VM_FAULT_NOPAGE as success

6. Tofu: mcctrl side MMU notifier and CQ/BCH cleanup

7. copy_user_ranges: copy straight_start of struct vm_range

8. mcctrl: abort on invalid addr in mcexec_transfer_image()

9. mcctrl: fix access to uninitialized usrdata->cpu_topology_list

10. mcexec: propagate error in __NR_gettid handler

11. mcexec_transfer_image(): map exact size of remote memory (instead of forcing PAGE_SIZE)

12. xpmem: fault stack area of remote process if VM range doesn’t yet exist

13. Tofu: fault stack area if VM range doesn’t exist in STAG registration

14. __mcctrl_os_read_write_cpu_register: fix timeout

15. mbind: Use range_policy’s numamask as priority on MPOL_BIND

16. migrate: Don’t migrate on in-kernel interrupt

17. Send a signal to mcexec after switching to that process.

18. uti: fix syscall response is mis-consumed by __do_in_kernel_irq_syscall

19. uti: fix handling UTI_CPU_SET env

20. do_execveat: kill instead of panic when init_process_stack fails

21. remote_page_fault is handled by the offloaded thread.

22. coredump: fix behavior when gencore fail

23. xpmem: truncates the size of xpmem_attach at the page boundary (workaround for fjmpi)

24. __mcctrl_os_read_write_cpu_register: spin timeout in mcctrl_ikc_send_wait()

Version 1.7.1 (Dec 23, 2020)

IHK major updates

1. d5d5c23 Tofu: support for barrier gate

2. Tofu: proper cleanup of premapped DMA regions

3. Tofu: initial version

4. SMP: try with GFP_ATOMIC as well in mem reserve

IHK major bug fixes

1. ihklib: ihk(_os)_query_{cpu,mem}: allow to pass empty array

2. SMP: non compound page free and GFP_ATOMIC

3. ihk_get_num_os_instances: don’t open /dev/mcdN

4. ihklib: ihk_create_os_str: fix variable prefix

McKernel major updates

1. stragiht map: creates a straight map covering the whole physical memory, and gives virtual address ranges out of it to maps to which physical pages are allocated on map

2. free-time, lazy, potentially Linux-side page-zeroing

3. Tofu built-in driver: supports memory registration and barrier gate setup

4. kmalloc cache

McKernel major bug fixes

1. mmap: return -EINVAL for non-anonymous, MAP_HUGETLB map

2. kernel: increase stack size

3. Tofu: proper cleanup of device files when mcexec gets killed

Version 1.7.0 (Nov 25, 2020)

IHK major updates

1. ihklib: add ihk_create_os_str

2. ihklib: ihk_reserve_mem: add capped best effort to balanced

IHK major bug fixes

1. make /dev/mcdN sharable

2. acpi: compat: RHEL-8.2

3. gic_chip_data: compat: RHEL-8.3

McKernel major updates

1. arm64: Contiguous PTE support

2. arm64: Scalable Vector Extension (SVE) support

3. arm64: PMU overflow interrupt support

4. arm64 port: Direct access to Mckernel memory from Linux

5. arm64 port: utility thread offloading, which spawns thread onto Linux CPU

6. eclair: support for live debug

7. Crash utility extension

8. Replace mcoverlayfs with a soft userspace overlay

9. Build system is switched to cmake

10. Core dump includes thread information

11. mcinspect and mcps: DWARF based LWK inspection

McKernel major bug fixes

1. shmobj: Fix rusage counting for large page

2. mcctrl control: task start_time changed to u64 nsec

3. mcctrl: add handling for one more level of page tables

4. Add kernel argument to turn on/off time sharing

5. flatten_string / process env: realign env and clear trailing bits

6. madvise: Add MADV_HUGEPAGE support

7. mcctrl: remove in-kernel calls to syscalls

8. arch_cpu_read_write_register: error return fix.

9. set_cputime(): interrupt enable/disable fix.

10. set_mempolicy(): Add mode check.

11. mbind(): Fix memory_range_lock deadlock.

12. ihk_ikc_recv: Record channel to packet for release

13. Add set_cputime() kernel to kernel case and mode enum.

14. execve: Call preempt_enable() before error-exit

15. memory/x86_64: fix linux safe_kernel_map

16. do_kill(): fix pids table when nr of threads is larger than num_processors

17. shmget: Use transparent huge pages when page size isn’t specified

18. prctl: Add support for PR_SET_THP_DISABLE and PR_GET_THP_DISABLE

19. monitor_init: fix undetected hang on highest numbered core

20. init_process_stack: change premapped stack size based on arch

21. x86 syscalls: add a bunch of XXat() delegated syscalls

22. do_pageout: fix direct kernel-user access

23. stack: add hwcap auxval

24. perf counters: add arch-specific perf counters

25. Added check of nohost to terminate_host().

26. kmalloc: Fix address order in free list

27. sysfs: use nr_cpu_ids for cpumasks (fixes libnuma parsing error on ARM)

28. monitor_init: Use ihk_mc_cpu_info()

29. Fix ThunderX2 write-combined PTE flag insanity

30. ARM: eliminate zero page mapping (i.e, init_low_area())

31. eliminate futex_cmpxchg_enabled check (not used and dereffed a NULL pointer)

32. page_table: Fix return value of lookup_pte when ptl4 is blank

33. sysfs: add missing symlinks for cpu/node

34. Make Linux handler run when mmap to procfs.

35. Separate mmap area from program loading (relocation) area

36. move rusage into kernel ELF image (avoid dynamic alloc before NUMA init)

37. arm: turn off cpu on panic

38. page fault handler: protect thread accesses

39. Register PPD and release_handler at the same time.

40. fix to missing exclusive processing between terminate() and finalize_process().

41. perfctr_stop: add flags to no ‘disable_intens’

42. fileobj, shmobj: free pages in object destructor (as opposed to page_unmap())

43. clear_range_l1, clear_range_middle: Fix handling contiguous PTE

44. do_mmap: don’t pre-populate the whole file when asked for smaller segment

45. invalidate_one_page: Support shmobj and contiguous PTE

46. ubsan: fix undefined shifts

47. x86: disable zero mapping and add a boot pt for ap trampoline

48. rusage: Don’t count PF_PATCH change

49. Fixed time processing.

50. copy_user_pte: vmap area not owned by McKernel

51. gencore: Zero-clear ELF header and memory range table

52. rpm: ignore CMakeCache.txt in dist and relax BuildRequires on cross build

53. gencore: Allocate ELF header to heap instead of stack

54. nanosleep: add cpu_pause() in spinwait loop

55. init_process: add missing initializations to proc struct

56. rus_vm_fault: always use a packet on the stack

57. process stack: use PAGE_SIZE in aux vector

58. copy_user_pte: base memobj copy on range & VR_PRIVATE

59. arm64: ptrace: Fix overwriting 1st argument with return value

60. page fault: use cow for private device mappings

61. reproductible builds: remove most install paths in c code

62. page fault: clear writable bit for non-dirtying access to shared ranges

63. mcreboot/mcstop+release: support for regular user execution

64. irqbalance_mck: replace extra service with service drop-in

65. do_mmap: give addr argument a chance even if not MAP_FIXED

66. x86: fix xchg() and cmpxchg() macros

67. IHK: support for using Linux work IRQ as IKC interrupt (optional)

68. MCS: fix ARM64 issue by using smp_XXX() functions (i.e., barrier()s)

69. procfs: add number of threads to stat and status

70. memory_range_lock: Fix deadlock in procfs/sysfs handler

71. flush instruction cache at context switch time if necessary

72. arm64: Fix PMU related functions

73. page_fault_process_memory_range: Disable COW for VM region with zeroobj

74. extend_process_region: Fall back to demand paging when not contiguous

75. munmap: fix deadlock with remote pagefault on vm range lock

76. procfs: if memory_range_lock fails, process later

77. migrate-cpu: Prevent migration target from calling schedule() twice

78. sched_request_migrate(): fix race condition between migration req and IRQs

79. get_one_cpu_topology: Renumber core_id (physical core id)

80. bb7e140 procfs cpuinfo: use sequence number as processor

81. set_host_vma(): do NOT read protect Linux VMA

82. hugefileobj: rewrite page allocation/handling

83. VM: use RW spinlock for vm_range_lock

84. /dev/shm: use Linux PFNs and populate mappings

85. Make struct ihk_os_rusage compatible with mckernel_rusage (workaround for Fugaku)

86. Record pthread routine address in clone(), keep helper threads on caller CPU core (workaround for Fugaku)

87. struct process: fix type of group_exit_status

88. tgkill: Fix argument validatation

89. set_robust_list: Add error check

90. mcexec: Don’t forward SIGTSTP SIGTTIN SIGTTOUT to mckernel

91. syscall: add prlimit64

92. stack: grow on page fault

93. mcexec: use FLIB_NUM_PROCESS_ON_NODE when -n not specified (Fugaku specific)

Version 1.6.0 (Nov 11, 2018)

McKernel major updates

1. McKernel and Linux share one unified kernel virtual address space. That is, McKernel sections resides in Linux sections spared for modules. In this way, Linux can access the McKernel kernel memory area.

2. hugetlbfs support

3. IHK is now included as a git submodule

4. Debug messages are turned on/off in per souce file basis at run-time.

5. It’s prohibited for McKernel to access physical memory ranges which Linux didn’t give to McKernel.

6. UTI (capability to spawn a thread on Linux CPU) improvement:

   • System calls issued from the thread are hooked by modifying binary in memory.

McKernel major bug fixes

#<digits> below denotes the redmine issue number (https://postpeta.pccluster.org/redmine/).

1. #926: shmget: Hide object with IPC_RMID from shmget

2. #1028: init_process: Inherit parent cpu_set

3. #995: Fix shebang recorded in argv[0]

4. #1024: Fix VMAP virtual address leak

5. #1109: init_process_stack: Support “ulimit -s unlimited”

6. x86 mem init: do not map identity mapping

7. mcexec_wait_syscall: requeue potential request on interrupted wait

8. mcctrl_ikc_send_wait: fix interrupt with do_frees == NULL

9. pager_req_read: handle short read

10. kprintf: only call eventfd() if it is safe to interrupt

11. process_procfs_request: Add Pid to /proc/<PID>/status

12. terminate: fix oversubscribe hang when waiting for other threads on same CPU to die

13. mcexec: Do not close fd returned to mckernel side

14. #976: execve: Clear sigaltstack and fp_regs

15. #1002: perf_event: Specify counter by bit_mask on start/stop

16. #1027: schedule: Don’t reschedule immediately when wake up on migrate

17. #mcctrl: lookup unexported symbols at runtime

18. __sched_wakeup_thread: Notify interrupt_exit() of re-schedule

19. futex_wait_queue_me: Spin-sleep when timeout and idle_halt is specified

20. #1167: ihk_os_getperfevent,setperfevent: Timeout IKC sent by mcctrl

21. devobj: fix object size (POSTK_DEBUG_TEMP_FIX_36)

22. mcctrl: remove rus page cache

23. #1021: procfs: Support multiple reads of e.g. /proc/*/maps

24. #1006: wait: Delay wake-up parent within switch context

25. #1164: mem: Check if phys-mem is within the range of McKernel memory

26. #1039: page_fault_process_memory_range: Remove ihk_mc_map_virtual for CoW of device map

27. partitioned execution: pass process rank to LWK

28. process/vm: implement access_ok()

29. spinlock: rewrite spinlock to use Linux ticket head/tail format

30. #986: Fix deadlock involving mmap_sem and memory_range_lock

31. Prevent one CPU from getting chosen by concurrent forks

32. #1009: check_signal: system call restart is done only once

33. #1176: syscall: the signal received during system call processing is not processed.

34. #1036 syscall_time: Handle by McKernel

35. #1165 do_syscall: Delegate system calls to the mcexec with the same pid

36. #1194 execve: Fix calling ptrace_report_signal after preemption is disabled

37. #1005 coredump: Exclude special areas

38. #1018 procfs: Fix pread/pwrite to procfs fail when specified size is bigger than 4MB

39. #1180 sched_setaffinity: Check migration after decrementing in_interrupt

40. #771, #1179, #1143 ptrace supports threads

41. #1189 procfs/do_fork: wait until procfs entries are registered

42. #1114 procfs: add ‘/proc/pid/stat’ to mckernel side and fix its comm

43. #1116 mcctrl procfs: check entry was returned before using it

44. #1167 ihk_os_getperfevent,setperfevent: Return -ETIME when IKC timeouts

45. mcexec/execve: fix shebangs handling

46. procfs: handle ‘comm’ on mckernel side

47. ihk_os_setperfevent: Return number of registered events

48. mcexec: fix terminating zero after readlink()

Version 1.5.1 (July 9, 2018)

McKernel major updates

Watchdog timer to detect hang of McKernel

mcexec prints out the following line to its stderr when a hang of McKernel is detected.

mcexec detected hang of McKernel

The watchdog timer is enabled by passing -i <timeout_in_sec> option to mcreboot.sh. <timeout_in_sec> specifies the interval of checking if McKernel is alive.

For example, specify -i 600 to detect the hang with 10 minutes interval:

mcreboot.sh -i 600

The detailed step of the hang detection is as follows.

1. mcexec acquires eventfd for notification from IHK and perform epoll() on it.

2. A daemon called ihkmond monitors the state of McKernel periodically with the interval specified by the -i option. It judges that McKernel is hanging and notifies mcexec by the eventfd if its state hasn’t changed since the last check.

McKernel major bug fixes

1. #1146: pager_req_map(): do not take mmap_sem if not needed

2. #1135: prepare_process_ranges_args_envs(): fix saving cmdline

3. #1144: fileobj/devobj: record path name

4. #1145: fileobj: use MCS locks for per-file page hash

5. #1076: mcctrl: refactor prepare_image into new generic ikc send&wait

6. #1072: execve: fix execve with oversubscribing

7. #1132: execve: use thread variable instead of cpu_local_var(current)

8. #1117: mprotect: do not set page table writable for cow pages

9. #1143: syscall wait4: add _WALL (POSTK_DEBUG_ARCH_DEP_44)

10. #1064: rusage: Fix initialization of rusage->num_processors

11. #1133: pager_req_unmap: Put per-process data at exit

12. #731: do_fork: Propagate error code returned by mcexec

13. #1149: execve: Reinitialize vm_regions’s map area on execve

14. #1065: procfs: Show file names in /proc/<PID>/maps

15. #1112: mremap: Fix type of size arguments (from ssize_t to size_t)

16. #1121: sched_getaffinity: Check arguments in the same order as in Linux

17. #1137: mmap, mremap: Check arguments in the same order as in Linux

18. #1122: fix return value of sched_getaffinity

19. #732: fix: /proc/<PID>/maps outputs a unnecessary NULL character

Version 1.5.0 (Apr 5, 2018)

McKernel major updates

1. Aid for Linux version migration: Detect /proc, /sys format change between two kernel verions

2. Swap out * Only swap-out anonymous pages for now

3. Improve support of /proc/maps

4. mcstat: Linux tool to show resource usage

McKernel major bug fixes

1. #727: execve: Fix memory leak when receiving SIGKILL

2. #829: perf_event_open: Support PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE

3. #906: mcexec: Check return code of fork()

4. #1038: mcexec: Timeout when incorrect value is given to -n option

5. #943 #945 #946 #960 #961: mcexec: Support strace

6. #1029: struct thread is not released with stress-test involving signal and futex

7. #863 #870 : Respond immediately to terminating signal when offloading system call

8. #1119: translate_rva_to_rpa(): use 2MB blocks in 1GB pages on x86

9. #898: Shutdown OS only after no in-flight IKC exist

10. #882: release_handler: Destroy objects as the process which opened it

11. #882: mcexec: Make child process exit if the parent is killed during fork()

12. #925: XPMEM: Don’t destroy per-process object of the parent

13. #885: ptrace: Support the case where a process attaches its child

14. #1031: sigaction: Support SA_RESETHAND

15. #923: rus_vm_fault: Return error when a thread not performing system call offloading causes remote page fault

16. #1032 #1033 #1034: getrusage: Fix ru_maxrss, RUSAGE_CHILDREN, ru_stime related bugs

17. #1120: getrusage: Fix deadlock on thread->times_update

18. #1123: Fix deadlock related to wait_queue_head_list_node

19. #1124: Fix deadlock of calling terminate() from terminate()

20. #1125: Fix deadlock related to thread status

    • Related functions are: hold_thread(), do_kill() and terminate()

21. #1126: uti: Fix uti thread on the McKernel side blocks others in do_syscall()

22. #1066: procfs: Show Linux /proc/self/cgroup

23. #1127: prepare_process_ranges_args_envs(): fix generating saved_cmdline to avoid PF in strlen()

24. #1128: ihk_mc_map/unmap_virtual(): do proper TLB invalidation

25. #1043: terminate(): fix update_lock and threads_lock order to avoid deadlock

26. #1129: mcreboot.sh: Save /proc/irq/*/smp_affinity to /tmp/mcreboot

27. #1130: mcexec: drop READ_IMPLIES_EXEC from personality

McKernel workarounds

1. Forbid CPU oversubscription

   • It can be turned on by mcreboot.sh -O option

Version 1.4.0 (Oct 30, 2017)

Abstracted event type support in perf_event_open()

PERF_TYPE_HARDWARE and PERF_TYPE_CACHE types are supported.

Direct user-space access

Code lines using direct user-space access (e.g. passing user-space pointer to memcpy()) becomes more portable across processor architectures. The modification follows the following rules.

1. Move the code section as it is to the architecture dependent directory if it is a part of the critical-path.

2. Otherwise, rewrite the code section by using the portable methods. The methods include copy_from_user(), copy_to_user(), pte_get_phys() and phys_to_virt().

MPI and OpenMP micro-bench tests

The performance figures of MPI and OpenMP primitives are compared with those of Linux by using Intel MPI Benchmarks and EPCC OpenMP Micro Benchmark.

Version 1.3.0 (Sep 30, 2017)

Kernel dump

1. A dump level of “only kernel memory” is added.

The following two levels are available now:

0	Dump all
24	Dump only kernel memory

The dump level can be set by -d option in ihkosctl or the argument for ihk_os_makedumpfile(), as shown in the following examples:

Command:             ihkosctl 0 dump -d 24
Function call:       ihk_os_makedumpfile(0, NULL, 24, 0);

1. Dump file is created when Linux panics.

The dump level can be set by dump_level kernel argument, as shown in the following example:

ihkosctl 0 kargs "hidos dump_level=24"

The IHK dump function is registered to panic_notifier_list when creating /dev/mcdX and called when Linux panics.

Quick Process Launch

MPI process launch time and some of the initialization time can be reduced in application consisting of multiple MPI programs which are launched in turn in the job script.

The following two steps should be performed to use this feature: #. Replace mpiexec with ql_mpiexec_start and add some lines for ql_mpiexec_finalize in the job script #. Modify the app so that it can repeat calculations and wait for the instructions from ql_mpiexec_{start,finalize} at the end of the loop

The first step is explained using an example. Assume the original job script looks like this:

/* Execute ensamble simulation and then data assimilation, and repeat this ten times */
for i in {1..10}; do

   /* Each ensamble simulation execution uses 100 nodes, launch ten of them in parallel */
   for j in {1..10}; do
      mpiexec -n 100 -machinefile ./list1_$j p1.out a1 & pids[$i]=$!;
   done

   /* Wait until the ten ensamble simulation programs finish */
   for j in {1..10}; do wait ${pids[$j]}; done

   /* Launch one data assimilation program using 1000 nodes */
   mpiexec -n 1000 -machinefile ./list2 p2.out a2
done

The job script should be modified like this:

for i in {1..10}; do
   for j in {1..10}; do
      /*  Replace mpiexec with ql_mpiexec_start */
      ql_mpiexec_start -n 100 -machinefile ./list1_$j p1.out a1 & pids[$j]=$!;
   done

   for j in {1..10}; do wait ${pids[$j]}; done

   ql_mpiexec_start -n 1000 -machinefile ./list2 p2.out a2
done

/* p1.out and p2.out don't exit but are waiting for the next calculation. So tell them to exit */
for j in {1..10}; do
   ql_mpiexec_finalize -machinefile ./list1_$i p1.out a1;
done

ql_mpiexec_finalize -machinefile ./list2 p2.out a2;

The second step is explained using a pseudo-code.

MPI_Init();
Prepare data exchange with preceding / following MPI programs
loop:
foreach Fortran module
   Initialize data using command-line argments, parameter files, environment variables
   Input data from preceding MPI programs / Read snap-shot
   Perform main calculation
   Output data to following MPI programs / Write snap-shot
   /* ql_client() waits for command of ql_mpiexec_{start,finish} */
   if (ql_client() == QL_CONTINUE) { goto loop; }
   MPI_Finalize();

qlmpilib.h should be included in the code and libql{mpi,fort}.so should be linked to the executable file.