IHK/McKernel

_images/mckernel-logo.png

IHK/McKernel is a light-weight multi-kernel operating system designed for high-end supercomputing. It runs Linux and McKernel, a light-weight kernel (LWK), side-by-side inside compute nodes and aims at the following:

  • Provide scalable and consistent execution of large-scale parallel scientific applications, but at the same time maintain the ability to rapidly adapt to new hardware features and emerging programming models

  • Provide efficient memory and device management so that resource contention and data movement are minimized at the system level

  • Eliminate OS noise by isolating OS services in Linux and provide jitter free execution on the LWK

  • Support the full POSIX/Linux APIs by selectively offloading (slow-path) system calls to Linux

See Quick Guide -- Installation for jump start.

1. Introduction

IHK/McKernel is a light-weight multi-kernel operating system designed for high-end supercomputing. It runs Linux and McKernel, a light-weight kernel (LWK), side-by-side inside compute nodes and aims at the following:

  • Provide scalable and consistent execution of large-scale parallel scientific applications, but at the same time maintain the ability to rapidly adapt to new hardware features and emerging programming models

  • Provide efficient memory and device management so that resource contention and data movement are minimized at the system level

  • Eliminate OS noise by isolating OS services in Linux and provide jitter free execution on the LWK

  • Support the full POSIX/Linux APIs by selectively offloading (slow-path) system calls to Linux

2. Background and Motivation

With the growing complexity of high-end supercomputers, the current system software stack faces significant challenges as we move forward to exascale and beyond. The necessity to deal with extreme degree of parallelism, heterogeneous architectures, multiple levels of memory hierarchy, power constraints, etc., advocates operating systems that can rapidly adapt to new hardware requirements, and that can support novel programming paradigms and runtime systems. On the other hand, a new class of more dynamic and complex applications are also on the horizon, with an increasing demand for application constructs such as in-situ analysis, workflows, elaborate monitoring and performance tools. This complexity relies not only on the rich features of POSIX, but also on the Linux APIs (such as the /proc, /sys filesystems, etc.) in particular.

2.1. Two Traditional HPC OS Approaches

Traditionally, light-weight operating systems specialized for HPC followed two approaches to tackle scalable execution of large-scale applications. In the full weight kernel (FWK) approach, a full Linux environment is taken as the basis, and features that inhibit attaining HPC scalability are removed, i.e., making it light-weight. The pure light-weight kernel (LWK) approach, on the other hand, starts from scratch and effort is undertaken to add sufficient functionality so that it provides a familiar API, typically something close to that of a general purpose OS, while at the same time it retains the desired scalability and reliability attributes. Neither of these approaches yields a fully Linux compatible environment.

2.2. The Multi-kernel Approach

A hybrid approach recognized recently by the system software community is to run Linux simultaneously with a lightweight kernel on compute nodes and multiple research projects are now pursuing this direction. The basic idea is that simulations run on an HPC tailored lightweight kernel, ensuring the necessary isolation for noiseless execution of parallel applications, but Linux is leveraged so that the full POSIX API is supported. Additionally, the small code base of the LWK can also facilitate rapid prototyping for new, exotic hardware features. Nevertheless, the questions of how to share node resources between the two types of kernels, where do device drivers execute, how exactly do the two kernels interact with each other and to what extent are they integrated, remain subjects of ongoing debate.

3. Architectural Overview

At the heart of the stack is a low-level software infrastructure called Interface for Heterogeneous Kernels (IHK). IHK is a general framework that provides capabilities for partitioning resources in a many-core environment (e.g.,CPU cores and physical memory) and it enables management of lightweight kernels. IHK can allocate and release host resources dynamically and no reboot of the host machine is required when altering configuration. IHK also provides a low-level inter-kernel messaging infrastructure, called the Inter-Kernel Communication (IKC) layer. An architectural overview of the main system components is shown below.

arch

McKernel is a lightweight kernel written from scratch. It is designed for HPC and is booted from IHK. McKernel retains a binary compatible ABI with Linux, however, it implements only a small set of performance sensitive system calls and the rest are offloaded to Linux. Specifically, McKernel has its own memory management, it supports processes and multi-threading with a simple round-robin cooperative (tick-less) scheduler, and it implements signaling. It also allows inter-process memory mappings and it provides interfaces to hardware performance counters.

3.1. Functionality

An overview of some of the principal functionalities of the IHK/McKernel stack is provided below.

3.1.1. System Call Offloading

System call forwarding in McKernel is implemented as follows. When an offloaded system call occurs, McKernel marshals the system call number along with its arguments and sends a message to Linux via a dedicated IKC channel. The corresponding proxy process running on Linux is by default waiting for system call requests through an ioctl() call into IHK’s system call delegator kernel module. The delegator kernel module’s IKC interrupt handler wakes up the proxy process, which returns to userspace and simply invokes the requested system call. Once it obtains the return value, it instructs the delegator module to send the result back to McKernel, which subsequently passes the value to user-space.

3.1.2. Unified Address Space

The unified address space model in IHK/McKernel ensures that offloaded system calls can seamlessly resolve arguments even in case of pointers. This mechanism is depicted below and is implemented as follows.

unified_ap

First, the proxy process is compiled as a position independent binary, which enables us to map the code and data segments specific to the proxy process to an address range which is explicitly excluded from McKernel’s user space. The grey box on the right side of the figure demonstrates the excluded region. Second, the entire valid virtual address range of McKernel’s application user-space is covered by a special mapping in the proxy process for which we use a pseudo file mapping in Linux. This mapping is indicated by the blue box on the left side of the figure.

4. Installation

The following OS distributions and platforms are recommended:

  • OS distribution

    • CentOS 7.3 or later

    • RHEL 7.3 or later

  • Platform

    • Intel Xeon

    • Intel Xeon Phi

    • Fujitsu A64FX

4.1. Prepare files for building McKernel

Grant read permission to the System.map file of your kernel version on the build machine:

sudo chmod a+r /boot/System.map-`uname -r`

Install the following packages to the build machine:

cmake kernel-devel binutils-devel systemd-devel numactl-devel gcc make nasm git libdwarf-devel

4.1.1. When having access to repositories

On RHEL 8, enable the CodeReady Linux Builder (CLB) repository:

sudo subscription-manager repos --enable codeready-builder-for-rhel-8-$(/bin/arch)-rpms

On CentOS 8, enable the PowerTools repository:

sudo dnf config-manager --set-enabled PowerTools

Install with yum:

sudo yum install cmake kernel-devel binutils-devel systemd-devel numactl-devel gcc make nasm git libdwarf-devel

4.1.2. When not having access to repositories

Ask the system administrator to install them. Note that libdwarf-devel is in the CodeReady Linux Builder repository on RHEL 8 or in the PowerTools repository on CentOS 8.

4.2. Clone, compile, install

Clone the source code:

mkdir -p ~/src/ihk+mckernel/
cd ~/src/ihk+mckernel/
git clone --recursive -b development https://github.com/RIKEN-SysSoft/mckernel.git

(Optional) Checkout to the specific branch or version:

cd mckernel
git checkout <pathspec>
git submodule update

Foe example, if you want to try the development branch, use “development” as the pathspec. If you want to try the prerelease version 1.7.0-0.2, use “1.7.0-0.2”.

Move to build directory:

mkdir -p ~/src/ihk+mckernel/build && cd ~/src/ihk+mckernel/build

Run cmake:

4.2.1. When not cross-compiling:

cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel ../mckernel

4.2.2. When cross-compiling:

cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel \
  -DUNAME_R=<target_uname_r> \
  -DKERNEL_DIR=<kernnel_dir> \
  -DBUILD_TARGET=smp-arm64 \
  -DCMAKE_TOOLCHAIN_FILE=../mckernel/cmake/cross-aarch64.cmake \
  ../mckernel

4.2.3. Install with cmake

Install with make:

make -j install

The kernel modules and McKernel kernel image should be installed under the ihk+mckernel folder in your home directory.

4.2.4. Install with rpm

Create the tarball and the spec file:

make dist
cp mckernel-<version>.tar.gz <rpmbuild>/SOURCES

Create the rpm package:

When not cross-compiling:
rpmbuild -ba scripts/mckernel.spec
When cross-compiling:
rpmbuild -ba scripts/mckernel.spec --target <target_uname_m> -D 'kernel_version <target_uname_r>' -D 'kernel_dir <kernel_source>'

Install the rpm package:

sudo rpm -ivh <rpmbuild>/RPMS/<arch>/mckernel-<version>-<release>_<linux_kernel_ver>_<dist>.<arch>.rpm

The kernel modules and McKernel kernel image are installed under the standard system directories.

4.3. Prepare files and change settings for installing McKernel

Disable SELinux of the compute nodes:

sudo vim /etc/selinux/config

Change the file to SELINUX=disabled. And then reboot the compute nodes:

sudo reboot

Install the following packages to the compute nodes:

systemd-libs numactl-libs libdwarf

4.3.1. When having access to repositories

On RHEL 8, enable the CodeReady Linux Builder (CLB) repository:

sudo subscription-manager repos --enable codeready-builder-for-rhel-8-$(/bin/arch)-rpms

On CentOS 8, enable the PowerTools repository:

sudo dnf config-manager --set-enabled PowerTools

Install with yum:

sudo yum install systemd-libs numactl-libs libdwarf

4.3.2. When not having access to repositories

Ask the system administrator to install them. Note that libdwarf is in the CodeReady Linux Builder repository on RHEL 8 or in the PowerTools repository on CentOS 8.

4.4. Boot McKernel

A boot script called mcreboot.sh is provided under sbin in the install folder. To boot on logical CPU 1 with 512MB of memory, use the following invocation:

export TOP=${HOME}/ihk+mckernel/
cd ${TOP}
sudo ./sbin/mcreboot.sh -c 1 -m 512m

You should see something similar like this if you display the McKernel’s kernel message log:

./sbin/ihkosctl 0 kmsg

IHK/McKernel started.
[ -1]: no_execute_available: 1
[ -1]: map_fixed: phys: 0xfee00000 => 0xffff860000009000 (1 pages)
[ -1]: setup_x86 done.
[ -1]: ns_per_tsc: 385
[ -1]: KCommand Line: hidos    dump_level=24
[ -1]: Physical memory: 0x1ad3000 - 0x21000000, 525520896 bytes, 128301 pages available @ NUMA: 0
[ -1]: NUMA: 0, Linux NUMA: 0, type: 1, available bytes: 525520896, pages: 128301
[ -1]: NUMA 0 distances: 0 (10),
[ -1]: map_fixed: phys: 0x28000 => 0xffff86000000a000 (2 pages)
[ -1]: Trampoline area: 0x28000
[ -1]: map_fixed: phys: 0x0 => 0xffff86000000c000 (1 pages)
[ -1]: # of cpus : 1
[ -1]: locals = ffff880001af6000
[  0]: BSP: 0 (HW ID: 1 @ NUMA 0)
[  0]: BSP: booted 0 AP CPUs
[  0]: Master channel init acked.
[  0]: vdso is enabled
IHK/McKernel booted.

4.5. Run a simple program on McKernel

The mcexec command line tool (which is also the Linux proxy process) can be used for executing applications on McKernel:

./bin/mcexec hostname
centos-vm

4.6. Shutdown McKernel

Finally, to shutdown McKernel and release CPU/memory resources back to Linux use the following command:

sudo ./sbin/mcstop+release.sh

5. The Team

The McKernel project was started at The University of Tokyo and currently it is mainly developed at RIKEN. Some of our collaborators include:

  • Hitachi

  • Fujitsu

  • CEA (France)

  • NEC

6. License

McKernel is GPL licensed, as found in the LICENSE file.

7. Contact

Please give your feedback to us via one of the following mailing lists. Subscription via www.pccluster.org is needed.

1. Architectural Overview

At the heart of the stack is a low-level software infrastructure called Interface for Heterogeneous Kernels (IHK). IHK is a general framework that provides capabilities for partitioning resources in a many-core environment (e.g.,CPU cores and physical memory) and it enables management of lightweight kernels. IHK can allocate and release host resources dynamically and no reboot of the host machine is required when altering configuration. IHK also provides a low-level inter-kernel messaging infrastructure, called the Inter-Kernel Communication (IKC) layer. An architectural overview of the main system components is shown below.

arch

McKernel is a lightweight kernel written from scratch. It is designed for HPC and is booted from IHK. McKernel retains a binary compatible ABI with Linux, however, it implements only a small set of performance sensitive system calls and the rest are offloaded to Linux. Specifically, McKernel has its own memory management, it supports processes and multi-threading with a simple round-robin cooperative (tick-less) scheduler, and it implements signaling. It also allows inter-process memory mappings and it provides interfaces to hardware performance counters.

1.1. Functionality

An overview of some of the principal functionalities of the IHK/McKernel stack is provided below.

1.1.1. System Call Offloading

System call forwarding in McKernel is implemented as follows. When an offloaded system call occurs, McKernel marshals the system call number along with its arguments and sends a message to Linux via a dedicated IKC channel. The corresponding proxy process running on Linux is by default waiting for system call requests through an ioctl() call into IHK’s system call delegator kernel module. The delegator kernel module’s IKC interrupt handler wakes up the proxy process, which returns to userspace and simply invokes the requested system call. Once it obtains the return value, it instructs the delegator module to send the result back to McKernel, which subsequently passes the value to user-space.

1.1.2. Unified Address Space

The unified address space model in IHK/McKernel ensures that offloaded system calls can seamlessly resolve arguments even in case of pointers. This mechanism is depicted below and is implemented as follows.

unified_ap

First, the proxy process is compiled as a position independent binary, which enables us to map the code and data segments specific to the proxy process to an address range which is explicitly excluded from McKernel’s user space. The grey box on the right side of the figure demonstrates the excluded region. Second, the entire valid virtual address range of McKernel’s application user-space is covered by a special mapping in the proxy process for which we use a pseudo file mapping in Linux. This mapping is indicated by the blue box on the left side of the figure.

2. Running Programs

2.1. Non-MPI programs

Insert mcexec before an executable:

mcexec ./a.out

2.2. MPI programs

Insert mcexec -n <processes-per-node> after mpirun and before an executable:

mpirun -n 32 mcexec -n 8 ./a.out

<processes-per-node> is the number of the processes per node and calculated by (number of MPI processes) / (number of nodes).

For example, <processes-per-node> equals to 4 (=32/8) when specifying the number of processes and nodes as follows with Fujitsu Technical Computing Suite.

#PJM --mpi "proc=32"
#PJM -L "node=8"

3. Limitations

  1. Pseudo devices such as /dev/mem and /dev/zero are not mmap()ed correctly even if the mmap() returns a success. An access of their mapping receives the SIGSEGV signal.

  2. clone() supports only the following flags. All the other flags cause clone() to return error or are simply ignored.

    • CLONE_CHILD_CLEARTID

    • CLONE_CHILD_SETTID

    • CLONE_PARENT_SETTID

    • CLONE_SETTLS

    • CLONE_SIGHAND

    • CLONE_VM

  3. PAPI has the following restriction.

    • Number of counters a user can use at the same time is up to the number of the physical counters in the processor.

  4. msync writes back only the modified pages mapped by the calling process.

  5. The following syscalls always return the ENOSYS error.

    • migrate_pages()

    • move_pages()

    • set_robust_list()

  6. The following syscalls always return the EOPNOTSUPP error.

    • arch_prctl(ARCH_SET_GS)

    • signalfd()

  7. signalfd4() returns a fd, but signal is not notified through the fd.

  8. set_rlimit sets the limit values but they are not enforced.

  9. Address randomization is not supported.

  10. brk() extends the heap more than requestd when -h (–extend-heap-by=) option of mcexec is used with the value larger than 4 KiB. syscall_pwrite02 of LTP would fail for this reason. This is because the test expects that the end of the heap is set to the same address as the argument of sbrk() and expects a segmentation violation occurs when it tries to access the memory area right next to the boundary. However, the optimization sets the end to a value larger than the requested. Therefore, the expected segmentation violation doesn’t occur.

  11. setpriority()/getpriority() won’t work. They might set/get the priority of a random mcexec thread. This is because there’s no fixed correspondence between a McKernel thread which issues the system call and a mcexec thread which handles the offload request.

  12. mbind() can set the policy but it is not used when allocating physical pages.

  13. MPOL_F_RELATIVE_NODES and MPOL_INTERLEAVE flags for set_mempolicy()/mbind() are not supported.

  14. The MPOL_BIND policy for set_mempolicy()/mbind() works as the same as the MPOL_PREFERRED policy. That is, the physical page allocator doesn’t give up the allocation when the specified nodes are running out of pages but continues to search pages in the other nodes.

  15. Kernel dump on Linux panic requires Linux kernel CentOS-7.4 and later. In addition, crash_kexec_post_notifiers kernel argument must be given to Linux kernel.

  16. setfsuid()/setfsgid() cannot change the id of the calling thread. Instead, it changes that of the mcexec worker thread which takes the system-call offload request.

  17. mmap (hugeTLBfs): The physical pages corresponding to a map are released when no McKernel process exist. The next map gets fresh physical pages.

  18. Sticky bit on executable file has no effect.

  19. Linux (RHEL-7 for x86_64) could hang when offlining CPUs in the process of booting McKernel due to the Linux bug, found in Linux-3.10 and fixed in the later version. One way to circumvent this is to always assign the same CPU set to McKernel.

  20. madvise:

    • MADV_HWPOISON and MADV_SOFT_OFFLINE always returns -EPERM.

    • MADV_MERGEABLE and MADV_UNMERGEABLE always returns -EINVAL.

    • MADV_HUGEPAGE and MADV_NOHUGEPAGE on file map returns -EINVAL except on RHEL-8 for aarch64.

  21. brk() and mmap() doesn’t report out-of-memory through its return value. Instead, page-fault reports the error.

  22. Anonymous mmap pre-maps requested number of pages when contiguous pages are available. Demand paging is used when not available.

  23. Mixing page sizes in anonymous shared mapping is not allowed. mmap creates vm_range with one page size. And munmap or mremap that needs the reduced page size changes the sizes of all the pages of the vm_range.

  24. ihk_os_getperfevent() could time-out when invoked from Fujitsu TCS (job-scheduler).

  25. The behaviors of madvise and mbind are changed to do nothing and report success as a workaround for Fugaku.

  26. mmap() allows unlimited overcommit. Note that it corresponds to setting sysctl vm.overcommit_memory to 1.

1. Installation

The following OS distributions and platforms are recommended:

  • OS distribution

    • CentOS 7.3 or later

    • RHEL 7.3 or later

  • Platform

    • Intel Xeon

    • Intel Xeon Phi

    • Fujitsu A64FX

1.1. Prepare files for building McKernel

Grant read permission to the System.map file of your kernel version on the build machine:

sudo chmod a+r /boot/System.map-`uname -r`

Install the following packages to the build machine:

cmake kernel-devel binutils-devel systemd-devel numactl-devel gcc make nasm git libdwarf-devel

1.1.1. When having access to repositories

On RHEL 8, enable the CodeReady Linux Builder (CLB) repository:

sudo subscription-manager repos --enable codeready-builder-for-rhel-8-$(/bin/arch)-rpms

On CentOS 8, enable the PowerTools repository:

sudo dnf config-manager --set-enabled PowerTools

Install with yum:

sudo yum install cmake kernel-devel binutils-devel systemd-devel numactl-devel gcc make nasm git libdwarf-devel

1.1.2. When not having access to repositories

Ask the system administrator to install them. Note that libdwarf-devel is in the CodeReady Linux Builder repository on RHEL 8 or in the PowerTools repository on CentOS 8.

1.2. Clone, compile, install

Clone the source code:

mkdir -p ~/src/ihk+mckernel/
cd ~/src/ihk+mckernel/
git clone --recursive -b development https://github.com/RIKEN-SysSoft/mckernel.git

(Optional) Checkout to the specific branch or version:

cd mckernel
git checkout <pathspec>
git submodule update

Foe example, if you want to try the development branch, use “development” as the pathspec. If you want to try the prerelease version 1.7.0-0.2, use “1.7.0-0.2”.

Move to build directory:

mkdir -p ~/src/ihk+mckernel/build && cd ~/src/ihk+mckernel/build

Run cmake:

1.2.1. When not cross-compiling:

cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel ../mckernel

1.2.2. When cross-compiling:

cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel \
  -DUNAME_R=<target_uname_r> \
  -DKERNEL_DIR=<kernnel_dir> \
  -DBUILD_TARGET=smp-arm64 \
  -DCMAKE_TOOLCHAIN_FILE=../mckernel/cmake/cross-aarch64.cmake \
  ../mckernel

1.2.3. Install with cmake

Install with make:

make -j install

The kernel modules and McKernel kernel image should be installed under the ihk+mckernel folder in your home directory.

1.2.4. Install with rpm

Create the tarball and the spec file:

make dist
cp mckernel-<version>.tar.gz <rpmbuild>/SOURCES

Create the rpm package:

When not cross-compiling:
rpmbuild -ba scripts/mckernel.spec
When cross-compiling:
rpmbuild -ba scripts/mckernel.spec --target <target_uname_m> -D 'kernel_version <target_uname_r>' -D 'kernel_dir <kernel_source>'

Install the rpm package:

sudo rpm -ivh <rpmbuild>/RPMS/<arch>/mckernel-<version>-<release>_<linux_kernel_ver>_<dist>.<arch>.rpm

The kernel modules and McKernel kernel image are installed under the standard system directories.

1.3. Prepare files and change settings for installing McKernel

Disable SELinux of the compute nodes:

sudo vim /etc/selinux/config

Change the file to SELINUX=disabled. And then reboot the compute nodes:

sudo reboot

Install the following packages to the compute nodes:

systemd-libs numactl-libs libdwarf

1.3.1. When having access to repositories

On RHEL 8, enable the CodeReady Linux Builder (CLB) repository:

sudo subscription-manager repos --enable codeready-builder-for-rhel-8-$(/bin/arch)-rpms

On CentOS 8, enable the PowerTools repository:

sudo dnf config-manager --set-enabled PowerTools

Install with yum:

sudo yum install systemd-libs numactl-libs libdwarf

1.3.2. When not having access to repositories

Ask the system administrator to install them. Note that libdwarf is in the CodeReady Linux Builder repository on RHEL 8 or in the PowerTools repository on CentOS 8.

1.4. Advanced: Enable Utility Thread offloading Interface (UTI)

UTI enables a runtime such as MPI runtime to spawn utility threads such as MPI asynchronous progress threads to Linux cores.

1.4.1. Install capstone

When compute nodes don't have access to repositories

Install EPEL capstone-devel:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo yum install capstone-devel
When compute nodes don't have access to repositories

Ask the system administrator to install capstone-devel. Note that it is in the EPEL repository.

1.4.2. Install syscall_intercept

git clone https://github.com/RIKEN-SysSoft/syscall_intercept.git
mkdir build && cd build
cmake <syscall_intercept>/arch/aarch64 -DCMAKE_INSTALL_PREFIX=<syscall-intercept-install> -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DTREAT_WARNINGS_AS_ERRORS=OFF

1.4.3. Install UTI for McKernel

Install:

git clone https://github.com/RIKEN-SysSoft/uti.git
mkdir build && cd build
../uti/configure --prefix=<mckernel-install> --with-rm=mckernel
make && make install

1.4.4. Install McKernel

Add -DENABLE_UTI=ON option to cmake:

CMAKE_PREFIX_PATH=<syscall-intercept-install> cmake -DCMAKE_INSTALL_PREFIX=${HOME}/ihk+mckernel -DENABLE_UTI=ON $HOME/src/ihk+mckernel/mckernel

1.4.5. Run programs

Add --enable-uti option to mcexec:

mcexec --enable-uti <command>

1.4.6. Install UTI for Linux

You should skip this step if it's already installed as with, for example, Fujitsu Technical Computing Suite.

Install by make
git clone https://github.com/RIKEN-SysSoft/uti.git
mkdir build && cd build
../uti/configure --prefix=<uti-install> --with-rm=linux
make && make install
Install by rpm
git clone https://github.com/RIKEN-SysSoft/uti.git
mkdir build && cd build
../uti/configure --prefix=<uti-install> --with-rm=linux
rm -f ~/rpmbuild/SOURCES/<version>.tar.gz
rpmbuild -ba ./scripts/uti.spec
rpm -Uvh uti-<version>-<release>-<arch>.rpm

2. Interface

3. Boot

4. Shutdown

1. External Specs

1.1. Overview

1.2. Function Specs

1.3. Command / Daemon Specs

2. Booting LWK

1. Interfaces

2. Interface details

Version 1.7.0rc4 (Apr 15, 2020)

McKernel major updates

  1. arm64: Contiguous PTE support

  2. arm64: Scalable Vector Extension (SVE) support

  3. arm64: PMU overflow interrupt support

  4. xpmem: Support large page attachment

  5. arm64 port: Direct access to Mckernel memory from Linux

  6. arm64 port: utility thread offloading, which spawns thread onto Linux CPU

  7. eclair: support for live debug

  8. Crash utility extension

  9. Replace mcoverlayfs with a soft userspace overlay

  10. Build system is switched to cmake

  11. Core dump includes thread information

McKernel major bug fixes

  1. shmobj: Fix rusage counting for large page

  2. mcctrl control: task start_time changed to u64 nsec

  3. mcctrl: add handling for one more level of page tables

  4. Add kernel argument to turn on/off time sharing

  5. flatten_string / process env: realign env and clear trailing bits

  6. madvise: Add MADV_HUGEPAGE support

  7. mcctrl: remove in-kernel calls to syscalls

  8. arch_cpu_read_write_register: error return fix.

  9. set_cputime(): interrupt enable/disable fix.

  10. set_mempolicy(): Add mode check.

  11. mbind(): Fix memory_range_lock deadlock.

  12. ihk_ikc_recv: Record channel to packet for release

  13. Add set_cputime() kernel to kernel case and mode enum.

  14. execve: Call preempt_enable() before error-exit

  15. memory/x86_64: fix linux safe_kernel_map

  16. do_kill(): fix pids table when nr of threads is larger than num_processors

  17. shmget: Use transparent huge pages when page size isn't specified

  18. prctl: Add support for PR_SET_THP_DISABLE and PR_GET_THP_DISABLE

  19. monitor_init: fix undetected hang on highest numbered core

  20. init_process_stack: change premapped stack size based on arch

  21. x86 syscalls: add a bunch of XXat() delegated syscalls

  22. do_pageout: fix direct kernel-user access

  23. stack: add hwcap auxval

  24. perf counters: add arch-specific perf counters

  25. Added check of nohost to terminate_host().

  26. kmalloc: Fix address order in free list

  27. sysfs: use nr_cpu_ids for cpumasks (fixes libnuma parsing error on ARM)

  28. monitor_init: Use ihk_mc_cpu_info()

  29. Fix ThunderX2 write-combined PTE flag insanity

  30. ARM: eliminate zero page mapping (i.e, init_low_area())

  31. eliminate futex_cmpxchg_enabled check (not used and dereffed a NULL pointer)

  32. page_table: Fix return value of lookup_pte when ptl4 is blank

  33. sysfs: add missing symlinks for cpu/node

  34. Make Linux handler run when mmap to procfs.

  35. Separate mmap area from program loading (relocation) area

  36. move rusage into kernel ELF image (avoid dynamic alloc before NUMA init)

  37. arm: turn off cpu on panic

  38. page fault handler: protect thread accesses

  39. Register PPD and release_handler at the same time.

  40. fix to missing exclusive processing between terminate() and finalize_process().

  41. perfctr_stop: add flags to no 'disable_intens'

  42. fileobj, shmobj: free pages in object destructor (as opposed to page_unmap())

  43. clear_range_l1, clear_range_middle: Fix handling contiguous PTE

  44. do_mmap: don't pre-populate the whole file when asked for smaller segment

  45. invalidate_one_page: Support shmobj and contiguous PTE

  46. ubsan: fix undefined shifts

  47. x86: disable zero mapping and add a boot pt for ap trampoline

  48. rusage: Don't count PF_PATCH change

  49. Fixed time processing.

  50. copy_user_pte: vmap area not owned by McKernel

  51. gencore: Zero-clear ELF header and memory range table

  52. rpm: ignore CMakeCache.txt in dist and relax BuildRequires on cross build

  53. gencore: Allocate ELF header to heap instead of stack

  54. nanosleep: add cpu_pause() in spinwait loop

  55. init_process: add missing initializations to proc struct

  56. rus_vm_fault: always use a packet on the stack

  57. process stack: use PAGE_SIZE in aux vector

  58. copy_user_pte: base memobj copy on range & VR_PRIVATE

  59. arm64: ptrace: Fix overwriting 1st argument with return value

  60. page fault: use cow for private device mappings

  61. reproductible builds: remove most install paths in c code

  62. page fault: clear writable bit for non-dirtying access to shared ranges

  63. mcreboot/mcstop+release: support for regular user execution

  64. irqbalance_mck: replace extra service with service drop-in

  65. do_mmap: give addr argument a chance even if not MAP_FIXED

  66. x86: fix xchg() and cmpxchg() macros

  67. IHK: support for using Linux work IRQ as IKC interrupt (optional)

  68. MCS: fix ARM64 issue by using smp_XXX() functions (i.e., barrier()s)

  69. procfs: add number of threads to stat and status

  70. memory_range_lock: Fix deadlock in procfs/sysfs handler

  71. flush instruction cache at context switch time if necessary

  72. arm64: Fix PMU related functions

  73. page_fault_process_memory_range: Disable COW for VM region with zeroobj

  74. extend_process_region: Fall back to demand paging when not contiguous

  75. munmap: fix deadlock with remote pagefault on vm range lock

  76. procfs: if memory_range_lock fails, process later

  77. migrate-cpu: Prevent migration target from calling schedule() twice

  78. sched_request_migrate(): fix race condition between migration req and IRQs

  79. get_one_cpu_topology: Renumber core_id (physical core id)

  80. bb7e140 procfs cpuinfo: use sequence number as processor

  81. set_host_vma(): do NOT read protect Linux VMA

Version 1.6.0 (Nov 11, 2018)

McKernel major updates

  1. McKernel and Linux share one unified kernel virtual address space. That is, McKernel sections resides in Linux sections spared for modules. In this way, Linux can access the McKernel kernel memory area.

  2. hugetlbfs support

  3. IHK is now included as a git submodule

  4. Debug messages are turned on/off in per souce file basis at run-time.

  5. It's prohibited for McKernel to access physical memory ranges which Linux didn't give to McKernel.

  6. UTI (capability to spawn a thread on Linux CPU) improvement:

    • System calls issued from the thread are hooked by modifying binary in memory.

McKernel major bug fixes

#<digits> below denotes the redmine issue number (https://postpeta.pccluster.org/redmine/).

  1. #926: shmget: Hide object with IPC_RMID from shmget

  2. #1028: init_process: Inherit parent cpu_set

  3. #995: Fix shebang recorded in argv[0]

  4. #1024: Fix VMAP virtual address leak

  5. #1109: init_process_stack: Support "ulimit -s unlimited"

  6. x86 mem init: do not map identity mapping

  7. mcexec_wait_syscall: requeue potential request on interrupted wait

  8. mcctrl_ikc_send_wait: fix interrupt with do_frees == NULL

  9. pager_req_read: handle short read

  10. kprintf: only call eventfd() if it is safe to interrupt

  11. process_procfs_request: Add Pid to /proc/<PID>/status

  12. terminate: fix oversubscribe hang when waiting for other threads on same CPU to die

  13. mcexec: Do not close fd returned to mckernel side

  14. #976: execve: Clear sigaltstack and fp_regs

  15. #1002: perf_event: Specify counter by bit_mask on start/stop

  16. #1027: schedule: Don't reschedule immediately when wake up on migrate

  17. #mcctrl: lookup unexported symbols at runtime

  18. __sched_wakeup_thread: Notify interrupt_exit() of re-schedule

  19. futex_wait_queue_me: Spin-sleep when timeout and idle_halt is specified

  20. #1167: ihk_os_getperfevent,setperfevent: Timeout IKC sent by mcctrl

  21. devobj: fix object size (POSTK_DEBUG_TEMP_FIX_36)

  22. mcctrl: remove rus page cache

  23. #1021: procfs: Support multiple reads of e.g. /proc/*/maps

  24. #1006: wait: Delay wake-up parent within switch context

  25. #1164: mem: Check if phys-mem is within the range of McKernel memory

  26. #1039: page_fault_process_memory_range: Remove ihk_mc_map_virtual for CoW of device map

  27. partitioned execution: pass process rank to LWK

  28. process/vm: implement access_ok()

  29. spinlock: rewrite spinlock to use Linux ticket head/tail format

  30. #986: Fix deadlock involving mmap_sem and memory_range_lock

  31. Prevent one CPU from getting chosen by concurrent forks

  32. #1009: check_signal: system call restart is done only once

  33. #1176: syscall: the signal received during system call processing is not processed.

  34. #1036 syscall_time: Handle by McKernel

  35. #1165 do_syscall: Delegate system calls to the mcexec with the same pid

  36. #1194 execve: Fix calling ptrace_report_signal after preemption is disabled

  37. #1005 coredump: Exclude special areas

  38. #1018 procfs: Fix pread/pwrite to procfs fail when specified size is bigger than 4MB

  39. #1180 sched_setaffinity: Check migration after decrementing in_interrupt

  40. #771, #1179, #1143 ptrace supports threads

  41. #1189 procfs/do_fork: wait until procfs entries are registered

  42. #1114 procfs: add '/proc/pid/stat' to mckernel side and fix its comm

  43. #1116 mcctrl procfs: check entry was returned before using it

  44. #1167 ihk_os_getperfevent,setperfevent: Return -ETIME when IKC timeouts

  45. mcexec/execve: fix shebangs handling

  46. procfs: handle 'comm' on mckernel side

  47. ihk_os_setperfevent: Return number of registered events

  48. mcexec: fix terminating zero after readlink()

Version 1.5.1 (July 9, 2018)

McKernel major updates

Watchdog timer to detect hang of McKernel

mcexec prints out the following line to its stderr when a hang of McKernel is detected.

mcexec detected hang of McKernel

The watchdog timer is enabled by passing -i <timeout_in_sec> option to mcreboot.sh. <timeout_in_sec> specifies the interval of checking if McKernel is alive.

For example, specify -i 600 to detect the hang with 10 minutes interval:

mcreboot.sh -i 600
The detailed step of the hang detection is as follows.

  1. mcexec acquires eventfd for notification from IHK and perform epoll() on it.

  2. A daemon called ihkmond monitors the state of McKernel periodically with the interval specified by the -i option. It judges that McKernel is hanging and notifies mcexec by the eventfd if its state hasn't changed since the last check.

McKernel major bug fixes

  1. #1146: pager_req_map(): do not take mmap_sem if not needed

  2. #1135: prepare_process_ranges_args_envs(): fix saving cmdline

  3. #1144: fileobj/devobj: record path name

  4. #1145: fileobj: use MCS locks for per-file page hash

  5. #1076: mcctrl: refactor prepare_image into new generic ikc send&wait

  6. #1072: execve: fix execve with oversubscribing

  7. #1132: execve: use thread variable instead of cpu_local_var(current)

  8. #1117: mprotect: do not set page table writable for cow pages

  9. #1143: syscall wait4: add _WALL (POSTK_DEBUG_ARCH_DEP_44)

  10. #1064: rusage: Fix initialization of rusage->num_processors

  11. #1133: pager_req_unmap: Put per-process data at exit

  12. #731: do_fork: Propagate error code returned by mcexec

  13. #1149: execve: Reinitialize vm_regions's map area on execve

  14. #1065: procfs: Show file names in /proc/<PID>/maps

  15. #1112: mremap: Fix type of size arguments (from ssize_t to size_t)

  16. #1121: sched_getaffinity: Check arguments in the same order as in Linux

  17. #1137: mmap, mremap: Check arguments in the same order as in Linux

  18. #1122: fix return value of sched_getaffinity

  19. #732: fix: /proc/<PID>/maps outputs a unnecessary NULL character

Version 1.5.0 (Apr 5, 2018)

McKernel major updates

  1. Aid for Linux version migration: Detect /proc, /sys format change between two kernel verions

  2. Swap out * Only swap-out anonymous pages for now

  3. Improve support of /proc/maps

  4. mcstat: Linux tool to show resource usage

McKernel major bug fixes

  1. #727: execve: Fix memory leak when receiving SIGKILL

  2. #829: perf_event_open: Support PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE

  3. #906: mcexec: Check return code of fork()

  4. #1038: mcexec: Timeout when incorrect value is given to -n option

  5. #943 #945 #946 #960 #961: mcexec: Support strace

  6. #1029: struct thread is not released with stress-test involving signal and futex

  7. #863 #870 : Respond immediately to terminating signal when offloading system call

  8. #1119: translate_rva_to_rpa(): use 2MB blocks in 1GB pages on x86

  9. #898: Shutdown OS only after no in-flight IKC exist

  10. #882: release_handler: Destroy objects as the process which opened it

  11. #882: mcexec: Make child process exit if the parent is killed during fork()

  12. #925: XPMEM: Don't destroy per-process object of the parent

  13. #885: ptrace: Support the case where a process attaches its child

  14. #1031: sigaction: Support SA_RESETHAND

  15. #923: rus_vm_fault: Return error when a thread not performing system call offloading causes remote page fault

  16. #1032 #1033 #1034: getrusage: Fix ru_maxrss, RUSAGE_CHILDREN, ru_stime related bugs

  17. #1120: getrusage: Fix deadlock on thread->times_update

  18. #1123: Fix deadlock related to wait_queue_head_list_node

  19. #1124: Fix deadlock of calling terminate() from terminate()

  20. #1125: Fix deadlock related to thread status

    • Related functions are: hold_thread(), do_kill() and terminate()

  21. #1126: uti: Fix uti thread on the McKernel side blocks others in do_syscall()

  22. #1066: procfs: Show Linux /proc/self/cgroup

  23. #1127: prepare_process_ranges_args_envs(): fix generating saved_cmdline to avoid PF in strlen()

  24. #1128: ihk_mc_map/unmap_virtual(): do proper TLB invalidation

  25. #1043: terminate(): fix update_lock and threads_lock order to avoid deadlock

  26. #1129: mcreboot.sh: Save /proc/irq/*/smp_affinity to /tmp/mcreboot

  27. #1130: mcexec: drop READ_IMPLIES_EXEC from personality

McKernel workarounds

  1. Forbid CPU oversubscription

    • It can be turned on by mcreboot.sh -O option

Version 1.4.0 (Oct 30, 2017)

Abstracted event type support in perf_event_open()

PERF_TYPE_HARDWARE and PERF_TYPE_CACHE types are supported.

Direct user-space access

Code lines using direct user-space access (e.g. passing user-space pointer to memcpy()) becomes more portable across processor architectures. The modification follows the following rules.

  1. Move the code section as it is to the architecture dependent directory if it is a part of the critical-path.

  2. Otherwise, rewrite the code section by using the portable methods. The methods include copy_from_user(), copy_to_user(), pte_get_phys() and phys_to_virt().

MPI and OpenMP micro-bench tests

The performance figures of MPI and OpenMP primitives are compared with those of Linux by using Intel MPI Benchmarks and EPCC OpenMP Micro Benchmark.

Version 1.3.0 (Sep 30, 2017)

Kernel dump

  1. A dump level of "only kernel memory" is added.

The following two levels are available now:

0

Dump all

24

Dump only kernel memory

The dump level can be set by -d option in ihkosctl or the argument for ihk_os_makedumpfile(), as shown in the following examples:

Command:             ihkosctl 0 dump -d 24
Function call:       ihk_os_makedumpfile(0, NULL, 24, 0);

  1. Dump file is created when Linux panics.

The dump level can be set by dump_level kernel argument, as shown in the following example:

ihkosctl 0 kargs "hidos dump_level=24"

The IHK dump function is registered to panic_notifier_list when creating /dev/mcdX and called when Linux panics.

Quick Process Launch

MPI process launch time and some of the initialization time can be reduced in application consisting of multiple MPI programs which are launched in turn in the job script.

The following two steps should be performed to use this feature: #. Replace mpiexec with ql_mpiexec_start and add some lines for ql_mpiexec_finalize in the job script #. Modify the app so that it can repeat calculations and wait for the instructions from ql_mpiexec_{start,finalize} at the end of the loop

The first step is explained using an example. Assume the original job script looks like this:

/* Execute ensamble simulation and then data assimilation, and repeat this ten times */
for i in {1..10}; do

   /* Each ensamble simulation execution uses 100 nodes, launch ten of them in parallel */
   for j in {1..10}; do
      mpiexec -n 100 -machinefile ./list1_$j p1.out a1 & pids[$i]=$!;
   done

   /* Wait until the ten ensamble simulation programs finish */
   for j in {1..10}; do wait ${pids[$j]}; done

   /* Launch one data assimilation program using 1000 nodes */
   mpiexec -n 1000 -machinefile ./list2 p2.out a2
done

The job script should be modified like this:

for i in {1..10}; do
   for j in {1..10}; do
      /*  Replace mpiexec with ql_mpiexec_start */
      ql_mpiexec_start -n 100 -machinefile ./list1_$j p1.out a1 & pids[$j]=$!;
   done

   for j in {1..10}; do wait ${pids[$j]}; done

   ql_mpiexec_start -n 1000 -machinefile ./list2 p2.out a2
done

/* p1.out and p2.out don't exit but are waiting for the next calculation. So tell them to exit */
for j in {1..10}; do
   ql_mpiexec_finalize -machinefile ./list1_$i p1.out a1;
done

ql_mpiexec_finalize -machinefile ./list2 p2.out a2;

The second step is explained using a pseudo-code.

MPI_Init();
Prepare data exchange with preceding / following MPI programs
loop:
foreach Fortran module
   Initialize data using command-line argments, parameter files, environment variables
   Input data from preceding MPI programs / Read snap-shot
   Perform main calculation
   Output data to following MPI programs / Write snap-shot
   /* ql_client() waits for command of ql_mpiexec_{start,finish} */
   if (ql_client() == QL_CONTINUE) { goto loop; }
   MPI_Finalize();

qlmpilib.h should be included in the code and libql{mpi,fort}.so should be linked to the executable file.