Blame README.md

Packit Service ac8aad
kpatch: dynamic kernel patching
Packit Service ac8aad
===============================
Packit Service ac8aad
Packit Service ac8aad
kpatch is a Linux dynamic kernel patching infrastructure which allows you to
Packit Service ac8aad
patch a running kernel without rebooting or restarting any processes.  It
Packit Service ac8aad
enables sysadmins to apply critical security patches to the kernel immediately,
Packit Service ac8aad
without having to wait for long-running tasks to complete, for users to log
Packit Service ac8aad
off, or for scheduled reboot windows.  It gives more control over uptime
Packit Service ac8aad
without sacrificing security or stability.
Packit Service ac8aad
Packit Service ac8aad
**WARNING: Use with caution!  Kernel crashes, spontaneous reboots, and data loss
Packit Service ac8aad
may occur!**
Packit Service ac8aad
Packit Service ac8aad
Here's a video of kpatch in action:
Packit Service ac8aad
Packit Service ac8aad
[![kpatch video](https://img.youtube.com/vi/juyQ5TsJRTA/0.jpg)](https://www.youtube.com/watch?v=juyQ5TsJRTA)
Packit Service ac8aad
Packit Service ac8aad
And a few more:
Packit Service ac8aad
Packit Service ac8aad
- https://www.youtube.com/watch?v=rN0sFjrJQfU
Packit Service ac8aad
- https://www.youtube.com/watch?v=Mftc80KyjA4
Packit Service ac8aad
Packit Service ac8aad
Installation
Packit Service ac8aad
------------
Packit Service ac8aad
Packit Service ac8aad
### Prerequisites
Packit Service ac8aad
Packit Service ac8aad
#### Fedora
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for compiling kpatch:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
UNAME=$(uname -r)
Packit Service ac8aad
sudo dnf install gcc kernel-devel-${UNAME%.*} elfutils elfutils-devel
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for the "kpatch-build" command:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
sudo dnf install pesign yum-utils openssl wget numactl-devel
Packit Service ac8aad
sudo dnf builddep kernel-${UNAME%.*}
Packit Service ac8aad
sudo dnf debuginfo-install kernel-${UNAME%.*}
Packit Service ac8aad
Packit Service ac8aad
# optional, but highly recommended
Packit Service ac8aad
sudo dnf install ccache
Packit Service ac8aad
ccache --max-size=5G
Packit Service ac8aad
Packit Service ac8aad
# optional, for kpatch-test
Packit Service ac8aad
sudo dnf install patchutils
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
#### RHEL 7
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for compiling kpatch:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
UNAME=$(uname -r)
Packit Service ac8aad
sudo yum install gcc kernel-devel-${UNAME%.*} elfutils elfutils-devel
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for the "kpatch-build" command:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
sudo yum-config-manager --enable rhel-7-server-optional-rpms
Packit Service ac8aad
sudo yum install pesign yum-utils zlib-devel \
Packit Service ac8aad
  binutils-devel newt-devel python-devel perl-ExtUtils-Embed \
Packit Service ac8aad
  audit-libs-devel numactl-devel pciutils-devel bison ncurses-devel
Packit Service ac8aad
Packit Service ac8aad
sudo yum-builddep kernel-${UNAME%.*}
Packit Service ac8aad
sudo debuginfo-install kernel-${UNAME%.*}
Packit Service ac8aad
Packit Service ac8aad
# optional, but highly recommended
Packit Service ac8aad
sudo yum install https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/c/ccache-3.3.4-1.el7.x86_64.rpm
Packit Service ac8aad
ccache --max-size=5G
Packit Service ac8aad
Packit Service ac8aad
# optional, for kpatch-test
Packit Service ac8aad
sudo yum install patchutils
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
#### CentOS 7
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for compiling kpatch:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
UNAME=$(uname -r)
Packit Service ac8aad
sudo yum install gcc kernel-devel-${UNAME%.*} elfutils elfutils-devel
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for the "kpatch-build" command:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
sudo yum install pesign yum-utils zlib-devel \
Packit Service ac8aad
  binutils-devel newt-devel python-devel perl-ExtUtils-Embed \
Packit Service ac8aad
  audit-libs audit-libs-devel numactl-devel pciutils-devel bison
Packit Service ac8aad
Packit Service ac8aad
# enable CentOS 7 debug repo
Packit Service ac8aad
sudo yum-config-manager --enable debug
Packit Service ac8aad
Packit Service ac8aad
sudo yum-builddep kernel-${UNAME%.*}
Packit Service ac8aad
sudo debuginfo-install kernel-${UNAME%.*}
Packit Service ac8aad
Packit Service ac8aad
# optional, but highly recommended - enable EPEL 7
Packit Service ac8aad
sudo yum install ccache
Packit Service ac8aad
ccache --max-size=5G
Packit Service ac8aad
Packit Service ac8aad
# optional, for kpatch-test
Packit Service ac8aad
sudo yum install patchutils
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
#### Oracle Linux 7
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for compiling kpatch:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
UNAME=$(uname -r)
Packit Service ac8aad
sudo yum install gcc kernel-devel-${UNAME%.*} elfutils elfutils-devel
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for the "kpatch-build" command:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
sudo yum install pesign yum-utils zlib-devel \
Packit Service ac8aad
  binutils-devel newt-devel python-devel perl-ExtUtils-Embed \
Packit Service ac8aad
  audit-libs numactl-devel pciutils-devel bison
Packit Service ac8aad
Packit Service ac8aad
# enable ol7_optional_latest repo
Packit Service ac8aad
sudo yum-config-manager --enable ol7_optional_latest
Packit Service ac8aad
Packit Service ac8aad
sudo yum-builddep kernel-${UNAME%.*}
Packit Service ac8aad
Packit Service ac8aad
# manually install kernel debuginfo packages
Packit Service ac8aad
rpm -ivh https://oss.oracle.com/ol7/debuginfo/kernel-debuginfo-$(uname -r).rpm
Packit Service ac8aad
rpm -ivh https://oss.oracle.com/ol7/debuginfo/kernel-debuginfo-common-x86_64-$(uname -r).rpm
Packit Service ac8aad
Packit Service ac8aad
# optional, but highly recommended - enable EPEL 7
Packit Service ac8aad
sudo yum install ccache
Packit Service ac8aad
ccache --max-size=5G
Packit Service ac8aad
Packit Service ac8aad
# optional, for kpatch-test
Packit Service ac8aad
sudo yum install patchutils
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
#### Ubuntu 14.04
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for compiling kpatch:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
apt-get install make gcc libelf-dev
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for the "kpatch-build" command:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
apt-get install dpkg-dev devscripts
Packit Service ac8aad
apt-get build-dep linux
Packit Service ac8aad
Packit Service ac8aad
# optional, but highly recommended
Packit Service ac8aad
apt-get install ccache
Packit Service ac8aad
ccache --max-size=5G
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install kernel debug symbols:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
# Add ddebs repository
Packit Service ac8aad
codename=$(lsb_release -sc)
Packit Service ac8aad
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
Packit Service ac8aad
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
Packit Service ac8aad
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
Packit Service ac8aad
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
Packit Service ac8aad
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
Packit Service ac8aad
EOF
Packit Service ac8aad
Packit Service ac8aad
# add APT key
Packit Service ac8aad
wget -Nq http://ddebs.ubuntu.com/dbgsym-release-key.asc -O- | sudo apt-key add -
Packit Service ac8aad
apt-get update && apt-get install linux-image-$(uname -r)-dbgsym
Packit Service ac8aad
```
Packit Service ac8aad
If there are no packages published yet to the codename-security pocket, the
Packit Service ac8aad
apt update may report a "404 Not Found" error, as well as a complaint about
Packit Service ac8aad
disabling the repository by default.  This message may be ignored (see issue
Packit Service ac8aad
#710).
Packit Service ac8aad
Packit Service ac8aad
#### Debian 9 (Stretch)
Packit Service ac8aad
Packit Service ac8aad
Since Stretch the stock kernel can be used without changes, however the
Packit Service ac8aad
version of kpatch in Stretch is too old so you still need to build it
Packit Service ac8aad
manually. Follow the instructions for Debian Jessie (next section) but skip
Packit Service ac8aad
building a custom kernel/rebooting.
Packit Service ac8aad
Packit Service ac8aad
#### Debian 8 (Jessie)
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for compiling kpatch:
Packit Service ac8aad
Packit Service ac8aad
    apt-get install make gcc libelf-dev build-essential
Packit Service ac8aad
Packit Service ac8aad
Install and prepare the kernel sources:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
apt-get install linux-source-$(uname -r)
Packit Service ac8aad
cd /usr/src && tar xvf linux-source-$(uname -r).tar.xz && ln -s linux-source-$(uname -r) linux && cd linux
Packit Service ac8aad
cp /boot/config-$(uname -r) .config
Packit Service ac8aad
for OPTION in CONFIG_KALLSYMS_ALL CONFIG_FUNCTION_TRACER ; do sed -i "s/# $OPTION is not set/$OPTION=y/g" .config ; done
Packit Service ac8aad
sed -i "s/^SUBLEVEL.*/SUBLEVEL =/" Makefile
Packit Service ac8aad
make -j`getconf _NPROCESSORS_CONF` deb-pkg KDEB_PKGVERSION=$(uname -r).9-1
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the kernel packages and reboot
Packit Service ac8aad
Packit Service ac8aad
    dpkg -i /usr/src/*.deb
Packit Service ac8aad
    reboot
Packit Service ac8aad
Packit Service ac8aad
Install the dependencies for the "kpatch-build" command:
Packit Service ac8aad
Packit Service ac8aad
    apt-get install dpkg-dev
Packit Service ac8aad
    apt-get build-dep linux
Packit Service ac8aad
Packit Service ac8aad
    # optional, but highly recommended
Packit Service ac8aad
    apt-get install ccache
Packit Service ac8aad
    ccache --max-size=5G
Packit Service ac8aad
Packit Service ac8aad
#### Debian 7 (Lenny)
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Add backports repositories:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
echo "deb http://http.debian.net/debian wheezy-backports main" > /etc/apt/sources.list.d/wheezy-backports.list
Packit Service ac8aad
echo "deb http://packages.incloudus.com backports-incloudus main" > /etc/apt/sources.list.d/incloudus.list
Packit Service ac8aad
wget http://packages.incloudus.com/incloudus/incloudus.pub -O- | apt-key add -
Packit Service ac8aad
aptitude update
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install the linux kernel, symbols and gcc 4.9:
Packit Service ac8aad
Packit Service ac8aad
    aptitude install -t wheezy-backports -y initramfs-tools
Packit Service ac8aad
    aptitude install -y gcc gcc-4.9 g++-4.9 linux-image-3.14 linux-image-3.14-dbg
Packit Service ac8aad
Packit Service ac8aad
Configure gcc 4.9 as the default gcc compiler:
Packit Service ac8aad
Packit Service ac8aad
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 20
Packit Service ac8aad
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 50
Packit Service ac8aad
    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.7 20
Packit Service ac8aad
    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 50
Packit Service ac8aad
Packit Service ac8aad
Install kpatch and these dependencies:
Packit Service ac8aad
Packit Service ac8aad
    aptitude install kpatch
Packit Service ac8aad
Packit Service ac8aad
Configure ccache (installed by kpatch package):
Packit Service ac8aad
Packit Service ac8aad
    ccache --max-size=5G
Packit Service ac8aad
Packit Service ac8aad
#### Gentoo
Packit Service ac8aad
Packit Service ac8aad
*NOTE: You'll need about 15GB of free disk space for the kpatch-build cache in
Packit Service ac8aad
`~/.kpatch` and for ccache.*
Packit Service ac8aad
Packit Service ac8aad
Install Kpatch and Kpatch dependencies:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
emerge --ask sys-kernel/kpatch
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Install ccache (optional):
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
emerge --ask dev-util/ccache
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
Configure ccache:
Packit Service ac8aad
Packit Service ac8aad
```bash
Packit Service ac8aad
ccache --max-size=5G
Packit Service ac8aad
```
Packit Service ac8aad
Packit Service ac8aad
### Build
Packit Service ac8aad
Packit Service ac8aad
Compile kpatch:
Packit Service ac8aad
Packit Service ac8aad
    make
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
### Install
Packit Service ac8aad
Packit Service ac8aad
OPTIONAL: Install kpatch to `/usr/local`:
Packit Service ac8aad
Packit Service ac8aad
    sudo make install
Packit Service ac8aad
Packit Service ac8aad
Alternatively, the kpatch and kpatch-build scripts can be run directly from the
Packit Service ac8aad
git tree.
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
Quick start
Packit Service ac8aad
-----------
Packit Service ac8aad
Packit Service ac8aad
> NOTE: While kpatch is designed to work with any recent Linux
Packit Service ac8aad
kernel on any distribution, the `kpatch-build` command has **ONLY** been tested
Packit Service ac8aad
and confirmed to work on Fedora 20 and later, RHEL 7, Oracle Linux 7, CentOS 7 and Ubuntu 14.04.
Packit Service ac8aad
Packit Service ac8aad
First, make a source code patch against the kernel tree using diff, git, or
Packit Service ac8aad
quilt.
Packit Service ac8aad
Packit Service ac8aad
As a contrived example, let's patch /proc/meminfo to show VmallocChunk in ALL
Packit Service ac8aad
CAPS so we can see it better:
Packit Service ac8aad
Packit Service ac8aad
    $ cat meminfo-string.patch
Packit Service ac8aad
    Index: src/fs/proc/meminfo.c
Packit Service ac8aad
    ===================================================================
Packit Service ac8aad
    --- src.orig/fs/proc/meminfo.c
Packit Service ac8aad
    +++ src/fs/proc/meminfo.c
Packit Service ac8aad
    @@ -95,7 +95,7 @@ static int meminfo_proc_show(struct seq_
Packit Service ac8aad
     		"Committed_AS:   %8lu kB\n"
Packit Service ac8aad
     		"VmallocTotal:   %8lu kB\n"
Packit Service ac8aad
     		"VmallocUsed:    %8lu kB\n"
Packit Service ac8aad
    -		"VmallocChunk:   %8lu kB\n"
Packit Service ac8aad
    +		"VMALLOCCHUNK:   %8lu kB\n"
Packit Service ac8aad
     #ifdef CONFIG_MEMORY_FAILURE
Packit Service ac8aad
     		"HardwareCorrupted: %5lu kB\n"
Packit Service ac8aad
     #endif
Packit Service ac8aad
Packit Service ac8aad
Build the patch module:
Packit Service ac8aad
Packit Service ac8aad
    $ kpatch-build -t vmlinux meminfo-string.patch
Packit Service ac8aad
    Using cache at /home/jpoimboe/.kpatch/3.13.10-200.fc20.x86_64/src
Packit Service ac8aad
    Testing patch file
Packit Service ac8aad
    checking file fs/proc/meminfo.c
Packit Service ac8aad
    Building original kernel
Packit Service ac8aad
    Building patched kernel
Packit Service ac8aad
    Detecting changed objects
Packit Service ac8aad
    Rebuilding changed objects
Packit Service ac8aad
    Extracting new and modified ELF sections
Packit Service ac8aad
    meminfo.o: changed function: meminfo_proc_show
Packit Service ac8aad
    Building patch module: kpatch-meminfo-string.ko
Packit Service ac8aad
    SUCCESS
Packit Service ac8aad
Packit Service ac8aad
> NOTE: The `-t vmlinux` option is used to tell `kpatch-build` to only look for
Packit Service ac8aad
> changes in the `vmlinux` base kernel image, which is much faster than also
Packit Service ac8aad
> compiling all the kernel modules.  If your patch affects a kernel module, you
Packit Service ac8aad
> can either omit this option to build everything, and have `kpatch-build`
Packit Service ac8aad
> detect which modules changed, or you can specify the affected kernel build
Packit Service ac8aad
> targets with multiple `-t` options.
Packit Service ac8aad
Packit Service ac8aad
That outputs a patch module named `kpatch-meminfo-string.ko` in the current
Packit Service ac8aad
directory.  Now apply it to the running kernel:
Packit Service ac8aad
Packit Service ac8aad
    $ sudo kpatch load kpatch-meminfo-string.ko
Packit Service ac8aad
    loading core module: /usr/local/lib/modules/3.13.10-200.fc20.x86_64/kpatch/kpatch.ko
Packit Service ac8aad
    loading patch module: kpatch-meminfo-string.ko
Packit Service ac8aad
Packit Service ac8aad
Done!  The kernel is now patched.
Packit Service ac8aad
Packit Service ac8aad
    $ grep -i chunk /proc/meminfo
Packit Service ac8aad
    VMALLOCCHUNK:   34359337092 kB
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
Patch Author Guide
Packit Service ac8aad
------------------
Packit Service ac8aad
Packit Service ac8aad
Unfortunately, live patching isn't always as easy as the previous example, and
Packit Service ac8aad
can have some major pitfalls if you're not careful.  To learn more about how to
Packit Service ac8aad
properly create live patches, see the [Patch Author
Packit Service ac8aad
Guide](doc/patch-author-guide.md).
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
How it works
Packit Service ac8aad
------------
Packit Service ac8aad
Packit Service ac8aad
kpatch works at a function granularity: old functions are replaced with new
Packit Service ac8aad
ones.  It has four main components:
Packit Service ac8aad
Packit Service ac8aad
- **kpatch-build**: a collection of tools which convert a source diff patch to
Packit Service ac8aad
  a patch module.  They work by compiling the kernel both with and without
Packit Service ac8aad
  the source patch, comparing the binaries, and generating a patch module
Packit Service ac8aad
  which includes new binary versions of the functions to be replaced.
Packit Service ac8aad
Packit Service ac8aad
- **patch module**: a kernel module (.ko file) which includes the
Packit Service ac8aad
  replacement functions and metadata about the original functions.
Packit Service ac8aad
Packit Service ac8aad
- **kpatch core module**: a kernel module (.ko file) which provides an
Packit Service ac8aad
  interface for the patch modules to register new functions for
Packit Service ac8aad
  replacement.  It uses the kernel ftrace subsystem to hook into the original
Packit Service ac8aad
  function's mcount call instruction, so that a call to the original function
Packit Service ac8aad
  is redirected to the replacement function.
Packit Service ac8aad
Packit Service ac8aad
- **kpatch utility:** a command-line tool which allows a user to manage a
Packit Service ac8aad
  collection of patch modules.  One or more patch modules may be
Packit Service ac8aad
  configured to load at boot time, so that a system can remain patched
Packit Service ac8aad
  even after a reboot into the same version of the kernel.
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
### kpatch-build
Packit Service ac8aad
Packit Service ac8aad
The "kpatch-build" command converts a source-level diff patch file to a kernel
Packit Service ac8aad
patch module.  Most of its work is performed by the kpatch-build script
Packit Service ac8aad
which uses a utility named `create-diff-object` to compare changed objects.
Packit Service ac8aad
Packit Service ac8aad
The primary steps in kpatch-build are:
Packit Service ac8aad
- Build the unstripped vmlinux for the kernel
Packit Service ac8aad
- Patch the source tree
Packit Service ac8aad
- Rebuild vmlinux and monitor which objects are being rebuilt.
Packit Service ac8aad
  These are the "changed objects".
Packit Service ac8aad
- Recompile each changed object with `-ffunction-sections -fdata-sections`,
Packit Service ac8aad
  resulting in the changed patched objects
Packit Service ac8aad
- Unpatch the source tree
Packit Service ac8aad
- Recompile each changed object with `-ffunction-sections -fdata-sections`,
Packit Service ac8aad
  resulting in the changed original objects
Packit Service ac8aad
- For every changed object, use `create-diff-object` to do the following:
Packit Service ac8aad
	* Analyze each original/patched object pair for patchability
Packit Service ac8aad
	* Add `.kpatch.funcs` and `.rela.kpatch.funcs` sections to the output object.
Packit Service ac8aad
	The kpatch core module uses this to determine the list of functions
Packit Service ac8aad
	that need to be redirected using ftrace.
Packit Service ac8aad
	* Add `.kpatch.dynrelas` and `.rela.kpatch.dynrelas` sections to the output object.
Packit Service ac8aad
	This will be used to resolve references to non-included local
Packit Service ac8aad
	and non-exported global symbols. These relocations will be resolved by the kpatch core module.
Packit Service ac8aad
	* Generate the resulting output object containing the new and modified sections
Packit Service ac8aad
- Link all the output objects into a cumulative object
Packit Service ac8aad
- Generate the patch module
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
### Patching
Packit Service ac8aad
Packit Service ac8aad
The patch modules register with the core module (`kpatch.ko`).
Packit Service ac8aad
They provide information about original functions that need to be replaced, and
Packit Service ac8aad
corresponding function pointers to the replacement functions.
Packit Service ac8aad
Packit Service ac8aad
The core module registers a handler function with ftrace.  The
Packit Service ac8aad
handler function is called by ftrace immediately before the original
Packit Service ac8aad
function begins executing.  This occurs with the help of the reserved mcount
Packit Service ac8aad
call at the beginning of every function, created by the gcc `-mfentry` flag.
Packit Service ac8aad
The ftrace handler then modifies the return instruction pointer (IP)
Packit Service ac8aad
address on the stack and returns to ftrace, which then restores the original
Packit Service ac8aad
function's arguments and stack, and "returns" to the new function.
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
Limitations
Packit Service ac8aad
-----------
Packit Service ac8aad
Packit Service ac8aad
- Patches which modify init functions (annotated with `__init`) are not
Packit Service ac8aad
  supported.  kpatch-build will return an error if the patch attempts
Packit Service ac8aad
  to do so.
Packit Service ac8aad
Packit Service ac8aad
- Patches which modify statically allocated data are not supported.
Packit Service ac8aad
  kpatch-build will detect that and return an error.  (In the future
Packit Service ac8aad
  we will add a facility to support it.  It will probably require the
Packit Service ac8aad
  user to write code which runs at patch module loading time which manually
Packit Service ac8aad
  updates the data.)
Packit Service ac8aad
Packit Service ac8aad
- Patches which change the way a function interacts with dynamically
Packit Service ac8aad
  allocated data might be safe, or might not.  It isn't possible for
Packit Service ac8aad
  kpatch-build to verify the safety of this kind of patch.  It's up to
Packit Service ac8aad
  the user to understand what the patch does, whether the new functions
Packit Service ac8aad
  interact with dynamically allocated data in a different way than the
Packit Service ac8aad
  old functions did, and whether it would be safe to atomically apply
Packit Service ac8aad
  such a patch to a running kernel.
Packit Service ac8aad
Packit Service ac8aad
- Patches which modify functions in vdso are not supported.  These run in
Packit Service ac8aad
  user-space and ftrace can't hook them.
Packit Service ac8aad
Packit Service ac8aad
- Patches which modify functions that are missing a `fentry` call are not
Packit Service ac8aad
  supported.  This includes any `lib-y` targets that are archived into a
Packit Service ac8aad
  `lib.a` library for later linking (for example, `lib/string.o`).
Packit Service ac8aad
Packit Service ac8aad
- Some incompatibilities currently exist between kpatch and usage of ftrace and
Packit Service ac8aad
  kprobes.  See the Frequently Asked Questions section for more details.
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
Frequently Asked Questions
Packit Service ac8aad
--------------------------
Packit Service ac8aad
Packit Service ac8aad
**Q. What's the relationship between kpatch and the upstream Linux live kernel
Packit Service ac8aad
patching component (livepatch)?**
Packit Service ac8aad
Packit Service ac8aad
Starting with Linux 4.0, the Linux kernel has livepatch, which is a new
Packit Service ac8aad
converged live kernel patching framework.  Livepatch is similar in
Packit Service ac8aad
functionality to the kpatch core module, though it doesn't yet have all the
Packit Service ac8aad
features that kpatch does.
Packit Service ac8aad
Packit Service ac8aad
kpatch-build already works with both livepatch and kpatch.  If your kernel has
Packit Service ac8aad
CONFIG\_LIVEPATCH enabled, it detects that and builds a patch module in the
Packit Service ac8aad
livepatch format.  Otherwise it builds a kpatch patch module.
Packit Service ac8aad
Packit Service ac8aad
The kpatch script also supports both patch module formats.
Packit Service ac8aad
Packit Service ac8aad
**Q. Isn't this just a virus/rootkit injection framework?**
Packit Service ac8aad
Packit Service ac8aad
kpatch uses kernel modules to replace code.  It requires the `CAP_SYS_MODULE`
Packit Service ac8aad
capability.  If you already have that capability, then you already have the
Packit Service ac8aad
ability to arbitrarily modify the kernel, with or without kpatch.
Packit Service ac8aad
Packit Service ac8aad
**Q. How can I detect if somebody has patched the kernel?**
Packit Service ac8aad
Packit Service ac8aad
When a patch module is loaded, the `TAINT_USER` or `TAINT_LIVEPATCH` flag is
Packit Service ac8aad
set.  (The latter flag was introduced in Linux version 4.0.)  To test for
Packit Service ac8aad
these flags, `cat /proc/sys/kernel/tainted` and check to see if the value of
Packit Service ac8aad
`TAINT_USER` (64) or `TAINT_LIVEPATCH` (32768) has been OR'ed in.
Packit Service ac8aad
Packit Service ac8aad
Note that the `TAINT_OOT_MODULE` flag (4096) will also be set, since the patch
Packit Service ac8aad
module is built outside the Linux kernel source tree.
Packit Service ac8aad
Packit Service ac8aad
If your patch module is unsigned, the `TAINT_FORCED_MODULE` flag (2) will also
Packit Service ac8aad
be set.  Starting with Linux 3.15, this will be changed to the more specific
Packit Service ac8aad
`TAINT_UNSIGNED_MODULE` (8192).
Packit Service ac8aad
Packit Service ac8aad
Linux versions starting with 4.9 also support a per-module `TAINT_LIVEPATCH`
Packit Service ac8aad
taint flag. This can be checked by verifying the output of
Packit Service ac8aad
`cat /sys/module/<kpatch module>/taint` -- a 'K' character indicates the
Packit Service ac8aad
presence of `TAINT_LIVEPATCH`.
Packit Service ac8aad
Packit Service ac8aad
**Q. Will it destabilize my system?**
Packit Service ac8aad
Packit Service ac8aad
No, as long as the patch is chosen carefully.  See the Limitations section
Packit Service ac8aad
above.
Packit Service ac8aad
Packit Service ac8aad
**Q. Why does kpatch use ftrace to jump to the replacement function instead of
Packit Service ac8aad
adding the jump directly?**
Packit Service ac8aad
Packit Service ac8aad
ftrace owns the first "call mcount" instruction of every kernel function.  In
Packit Service ac8aad
order to keep compatibility with ftrace, we go through ftrace rather than
Packit Service ac8aad
updating the instruction directly.  This approach also ensures that the code
Packit Service ac8aad
modification path is reliable, since ftrace has been doing it successfully for
Packit Service ac8aad
years.
Packit Service ac8aad
Packit Service ac8aad
**Q. Is kpatch compatible with \<insert kernel debugging subsystem here\>?**
Packit Service ac8aad
Packit Service ac8aad
We aim to be good kernel citizens and maintain compatibility.  A kpatch
Packit Service ac8aad
replacement function is no different than a function loaded by any other kernel
Packit Service ac8aad
module.  Each replacement function has its own symbol name and kallsyms entry,
Packit Service ac8aad
so it looks like a normal function to the kernel.
Packit Service ac8aad
Packit Service ac8aad
- **oops stack traces**: Yes.  If the replacement function is involved in an
Packit Service ac8aad
  oops, the stack trace will show the function and kernel module name of the
Packit Service ac8aad
  replacement function, just like any other kernel module function.  The oops
Packit Service ac8aad
  message will also show the taint flag (see the FAQ "How can I detect if
Packit Service ac8aad
  somebody has patched the kernel" for specifics).
Packit Service ac8aad
- **kdump/crash**: Yes.  Replacement functions are normal functions, so crash
Packit Service ac8aad
  will have no issues.
Packit Service ac8aad
- **ftrace**: Yes, but certain uses of ftrace which involve opening the
Packit Service ac8aad
  `/sys/kernel/debug/tracing/trace` file or using `trace-cmd record` can result
Packit Service ac8aad
  in a tiny window of time where a patch gets temporarily disabled.  Therefore
Packit Service ac8aad
  it's a good idea to avoid using ftrace on a patched system until this issue
Packit Service ac8aad
  is resolved.
Packit Service ac8aad
- **systemtap/kprobes**: Some incompatibilities exist.
Packit Service ac8aad
  - If you setup a kprobe module at the beginning of a function before loading
Packit Service ac8aad
    a kpatch module, and they both affect the same function, kprobes "wins"
Packit Service ac8aad
    until the kprobe has been unregistered.  This is tracked in issue
Packit Service ac8aad
    [#47](https://github.com/dynup/kpatch/issues/47).
Packit Service ac8aad
  - Setting a kretprobe before loading a kpatch module could be unsafe.  See
Packit Service ac8aad
    issue [#67](https://github.com/dynup/kpatch/issues/67).
Packit Service ac8aad
- **perf**: Yes.
Packit Service ac8aad
- **tracepoints**: Patches to a function which uses tracepoints will result in
Packit Service ac8aad
  the tracepoints being effectively disabled as long as the patch is applied.
Packit Service ac8aad
Packit Service ac8aad
**Q. Why not use something like kexec instead?**
Packit Service ac8aad
Packit Service ac8aad
If you want to avoid a hardware reboot, but are ok with restarting processes,
Packit Service ac8aad
kexec is a good alternative.
Packit Service ac8aad
Packit Service ac8aad
**Q. If an application can't handle a reboot, it's designed wrong.**
Packit Service ac8aad
Packit Service ac8aad
That's a good poi... [system reboots]
Packit Service ac8aad
Packit Service ac8aad
**Q. What changes are needed in other upstream projects?**
Packit Service ac8aad
Packit Service ac8aad
We hope to make the following changes to other projects:
Packit Service ac8aad
Packit Service ac8aad
- kernel:
Packit Service ac8aad
	- ftrace improvements to close any windows that would allow a patch to
Packit Service ac8aad
	  be inadvertently disabled
Packit Service ac8aad
Packit Service ac8aad
**Q. Is it possible to register a function that gets called atomically with
Packit Service ac8aad
`stop_machine` when the patch module loads and unloads?**
Packit Service ac8aad
Packit Service ac8aad
We do have plans to implement something like that.
Packit Service ac8aad
Packit Service ac8aad
**Q. What kernels are supported?**
Packit Service ac8aad
Packit Service ac8aad
kpatch needs gcc >= 4.8 and Linux >= 3.9.
Packit Service ac8aad
Packit Service ac8aad
**Q. Is it possible to remove a patch?**
Packit Service ac8aad
Packit Service ac8aad
Yes.  Just run `kpatch unload` which will disable and unload the patch module
Packit Service ac8aad
and restore the function to its original state.
Packit Service ac8aad
Packit Service ac8aad
**Q. Can you apply multiple patches?**
Packit Service ac8aad
Packit Service ac8aad
Yes, but to prevent any unexpected interactions between multiple patch modules,
Packit Service ac8aad
it's recommended that patch upgrades are cumulative, so that each patch is a
Packit Service ac8aad
superset of the previous patch.  This can be achieved by combining the new
Packit Service ac8aad
patch with the previous patch using `combinediff` before running
Packit Service ac8aad
`kpatch-build`.
Packit Service ac8aad
Packit Service ac8aad
**Q. Why did kpatch-build detect a changed function that wasn't touched by the
Packit Service ac8aad
source patch?**
Packit Service ac8aad
Packit Service ac8aad
There could be a variety of reasons for this, such as:
Packit Service ac8aad
Packit Service ac8aad
- The patch changed an inline function.
Packit Service ac8aad
- The compiler decided to inline a changed function, resulting in the outer
Packit Service ac8aad
  function getting recompiled.  This is common in the case where the inner
Packit Service ac8aad
  function is static and is only called once.
Packit Service ac8aad
Packit Service ac8aad
**Q. How do I patch a function which is always on the stack of at least one
Packit Service ac8aad
task, such as schedule(), sys_poll(), sys_select(), sys_read(),
Packit Service ac8aad
sys_nanosleep(), etc?**
Packit Service ac8aad
Packit Service ac8aad
- If you're sure it would be safe for the old function and the new function to
Packit Service ac8aad
  run simultaneously, use the `KPATCH_FORCE_UNSAFE` macro to skip the
Packit Service ac8aad
  activeness safety check for the function.  See `kmod/patch/kpatch-macros.h`
Packit Service ac8aad
  for more details.
Packit Service ac8aad
Packit Service ac8aad
**Q. Are patching of kernel modules supported?**
Packit Service ac8aad
Packit Service ac8aad
- Yes.
Packit Service ac8aad
Packit Service ac8aad
**Q. Can you patch out-of-tree modules?**
Packit Service ac8aad
Packit Service ac8aad
- Yes, though it's currently a bit of a manual process.  See this
Packit Service ac8aad
  [message](https://www.redhat.com/archives/kpatch/2015-June/msg00004.html) on
Packit Service ac8aad
  the kpatch mailing list for more information.
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
Get involved
Packit Service ac8aad
------------
Packit Service ac8aad
Packit Service ac8aad
If you have questions or feedback, join the #kpatch IRC channel on freenode and
Packit Service ac8aad
say hi.  We also have a [mailing list](https://www.redhat.com/mailman/listinfo/kpatch).
Packit Service ac8aad
Packit Service ac8aad
Contributions are very welcome.  Feel free to open issues or PRs on github.
Packit Service ac8aad
For big PRs, it's a good idea to discuss them first in github issues or on the
Packit Service ac8aad
[mailing list](https://www.redhat.com/mailman/listinfo/kpatch) before you write
Packit Service ac8aad
a lot of code.
Packit Service ac8aad
Packit Service ac8aad
Packit Service ac8aad
License
Packit Service ac8aad
-------
Packit Service ac8aad
Packit Service ac8aad
kpatch is under the GPLv2 license.
Packit Service ac8aad
Packit Service ac8aad
This program is free software; you can redistribute it and/or
Packit Service ac8aad
modify it under the terms of the GNU General Public License
Packit Service ac8aad
as published by the Free Software Foundation; either version 2
Packit Service ac8aad
of the License, or (at your option) any later version.
Packit Service ac8aad
Packit Service ac8aad
This program is distributed in the hope that it will be useful,
Packit Service ac8aad
but WITHOUT ANY WARRANTY; without even the implied warranty of
Packit Service ac8aad
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
Packit Service ac8aad
GNU General Public License for more details.
Packit Service ac8aad
Packit Service ac8aad
You should have received a copy of the GNU General Public License
Packit Service ac8aad
along with this program; if not, write to the Free Software
Packit Service ac8aad
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.