|
Packit |
b55c50 |
# kvdo
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
A pair of kernel modules which provide pools of deduplicated and/or compressed
|
|
Packit |
b55c50 |
block storage.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Background
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
VDO (which includes [kvdo](https://github.com/dm-vdo/kvdo) and
|
|
Packit |
b55c50 |
[vdo](https://github.com/dm-vdo/vdo)) is software that provides inline
|
|
Packit |
b55c50 |
block-level deduplication, compression, and thin provisioning capabilities for
|
|
Packit |
b55c50 |
primary storage. VDO installs within the Linux device mapper framework, where
|
|
Packit |
b55c50 |
it takes ownership of existing physical block devices and remaps these to new,
|
|
Packit |
b55c50 |
higher-level block devices with data-efficiency capabilities.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
Deduplication is a technique for reducing the consumption of storage resources
|
|
Packit |
b55c50 |
by eliminating multiple copies of duplicate blocks. Compression takes the
|
|
Packit |
b55c50 |
individual unique blocks and shrinks them with coding algorithms; these reduced
|
|
Packit |
b55c50 |
blocks are then efficiently packed together into physical blocks. Thin
|
|
Packit |
b55c50 |
provisioning manages the mapping from LBAs presented by VDO to where the data
|
|
Packit |
b55c50 |
has actually been stored, and also eliminates any blocks of all zeroes.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
With deduplication, instead of writing the same data more than once each
|
|
Packit |
b55c50 |
duplicate block is detected and recorded as a reference to the original
|
|
Packit |
b55c50 |
block. VDO maintains a mapping from logical block addresses (used by the
|
|
Packit |
b55c50 |
storage layer above VDO) to physical block addresses (used by the storage layer
|
|
Packit |
b55c50 |
under VDO). After deduplication, multiple logical block addresses may be mapped
|
|
Packit |
b55c50 |
to the same physical block address; these are called shared blocks and are
|
|
Packit |
b55c50 |
reference-counted by the software.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
With VDO's compression, multiple blocks (or shared blocks) are compressed with
|
|
Packit |
b55c50 |
the fast LZ4 algorithm, and binned together where possible so that multiple
|
|
Packit |
b55c50 |
compressed blocks fit within a 4 KB block on the underlying storage. Mapping
|
|
Packit |
b55c50 |
from LBA is to a physical block address and index within it for the desired
|
|
Packit |
b55c50 |
compressed data. All compressed blocks are individually reference counted for
|
|
Packit |
b55c50 |
correctness.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
Block sharing and block compression are invisible to applications using the
|
|
Packit |
b55c50 |
storage, which read and write blocks as they would if VDO were not
|
|
Packit |
b55c50 |
present. When a shared block is overwritten, a new physical block is allocated
|
|
Packit |
b55c50 |
for storing the new block data to ensure that other logical block addresses
|
|
Packit |
b55c50 |
that are mapped to the shared physical block are not modified.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
This public source release of VDO includes two kernel modules, and a set of
|
|
Packit |
b55c50 |
userspace tools for managing them. The "kvdo" module implements fine-grained
|
|
Packit |
b55c50 |
storage virtualization, thin provisioning, block sharing, and compression; the
|
|
Packit |
b55c50 |
"uds" module provides memory-efficient duplicate identification. The userspace
|
|
Packit |
b55c50 |
tools include a pair of python scripts, "vdo" for creating and managing VDO
|
|
Packit |
b55c50 |
volumes, and "vdostats" for extracting statistics from those volumes.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Documentation
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
- [RHEL8 VDO Documentation](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_storage/index)
|
|
Packit |
b55c50 |
- [RHEL7 VDO Integration Guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-integration)
|
|
Packit |
b55c50 |
- [RHEL7 VDO Evaluation Guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-evaluation)
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Releases
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
Each branch on this project is intended to work with a specific release of
|
|
Packit |
b55c50 |
Enterprise Linux (Red Hat Enterprise Linux, CentOS, etc.). We try to maintain
|
|
Packit |
b55c50 |
compatibility with active Fedora releases, but some modifications may be
|
|
Packit |
b55c50 |
required.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
Version | Intended Enterprise Linux Release | Supported With Modifications
|
|
Packit |
b55c50 |
------- | --------------------------------- | -------------------------------
|
|
Packit |
b55c50 |
6.1.x.x | EL7 (3.10.0-*.el7) |
|
|
Packit |
b55c50 |
6.2.x.x | EL8 (4.18.0-*.el8) | Fedora 28, Fedora 29, Fedora 30, Rawhide
|
|
Packit |
b55c50 |
* Pre-built versions with the required modifications for the referenced Fedora
|
|
Packit |
b55c50 |
releases can be found
|
|
Packit |
b55c50 |
[here](https://copr.fedorainfracloud.org/coprs/rhawalsh/dm-vdo) and can be
|
|
Packit |
b55c50 |
used by running `dnf copr enable rhawalsh/dm-vdo`.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Status
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
VDO was originally developed by Permabit Technology Corp. as a proprietary set
|
|
Packit |
b55c50 |
of kernel modules and userspace tools. This software and technology has been
|
|
Packit |
b55c50 |
acquired by Red Hat, has been relicensed under the GPL (v2 or later), and this
|
|
Packit |
b55c50 |
repository begins the process of preparing for integration with the upstream
|
|
Packit |
b55c50 |
kernel.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
While this software has been relicensed there are a number of issues that must
|
|
Packit |
b55c50 |
still be addressed to be ready for upstream. These include:
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
- Conformance with kernel coding standards
|
|
Packit |
b55c50 |
- Use of existing EXPORT_SYMBOL_GPL kernel interfaces where appropriate
|
|
Packit |
b55c50 |
- Refactoring of primitives (e.g. cryptographic) to appropriate kernel
|
|
Packit |
b55c50 |
subsystems
|
|
Packit |
b55c50 |
- Support for non-x86-64 platforms
|
|
Packit |
b55c50 |
- Refactoring of platform layer abstractions and other changes requested by
|
|
Packit |
b55c50 |
upstream maintainers
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
We expect addressing these issues to take some time. In the meanwhile, this
|
|
Packit |
b55c50 |
project allows interested parties to begin using VDO immediately. The
|
|
Packit |
b55c50 |
technology itself is thoroughly tested, mature, and in production use since
|
|
Packit |
b55c50 |
2014 in its previous proprietary form.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Building
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
In order to build the kernel modules, invoke the following command
|
|
Packit |
b55c50 |
from the top directory of this tree:
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
make -C /usr/src/kernels/`uname -r` M=`pwd`
|
|
Packit |
b55c50 |
|
|
Packit Service |
4cccae |
* Patched sources that work with the most recent upstream kernels can be found
|
|
Packit Service |
4cccae |
[here](https://github.com/rhawalsh/kvdo).
|
|
Packit Service |
4cccae |
|
|
Packit |
b55c50 |
## Communication channels
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
Community feedback, participation and patches are welcome to the
|
|
Packit |
b55c50 |
vdo-devel@redhat.com mailing list -- subscribe
|
|
Packit |
b55c50 |
[here](https://www.redhat.com/mailman/listinfo/vdo-devel).
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Contributing
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
This project is currently a stepping stone towards integration with the Linux
|
|
Packit |
b55c50 |
kernel. As such, contributions are welcome via a process similar to that for
|
|
Packit |
b55c50 |
Linux kernel development. Patches should be submitted to the
|
|
Packit |
b55c50 |
vdo-devel@redhat.com mailing list, where they will be considered for
|
|
Packit |
b55c50 |
inclusion. This project does not accept pull requests.
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
## Licensing
|
|
Packit |
b55c50 |
|
|
Packit |
b55c50 |
[GPL v2.0 or later](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html).
|
|
Packit |
b55c50 |
All contributions retain ownership by their original author, but must also
|
|
Packit |
b55c50 |
be licensed under the GPL 2.0 or later to be merged.
|
|
Packit |
b55c50 |
|