.. contents:: Table of contents :backlinks: none Overview -------- Kconfiglib is a `Kconfig `__ implementation in Python 2/3. It started out as a helper library, but now has a enough functionality to also work well as a standalone Kconfig implementation (including `menuconfig interfaces `_ and `Kconfig extensions`_). The entire library is contained in `kconfiglib.py `_. The bundled scripts are implemented on top of it. Implementing your own scripts should be relatively easy, if needed. Kconfiglib is used exclusively by e.g. the `Zephyr `_ and `ACRN `_ projects. It is also used for many small helper scripts in various projects. Since Kconfiglib is based around a library, it can be used e.g. to generate a `Kconfig cross-reference `_ (note: heavy page), using the same robust Kconfig parser used for other Kconfig tools, instead of brittle ad-hoc parsing. The documentation generation script can be found `here `__. Kconfiglib implements the recently added `Kconfig preprocessor `_. For backwards compatibility, environment variables can be referenced both as ``$(FOO)`` (the new syntax) and as ``$FOO`` (the old syntax). The old syntax is deprecated, but will probably be supported for a very long time (the major version would be increased if support is ever dropped). Using the old syntax with an undefined environment variable keeps the string as is. Note: See `this issue `_ if you run into a "macro expanded to blank string" error with kernel 4.18+. Installation ------------ Installation with pip ~~~~~~~~~~~~~~~~~~~~~ Kconfiglib is available on `PyPI `_ and can be installed with e.g. .. code:: $ pip(3) install kconfiglib Microsoft Windows is supported. The ``pip`` installation will give you both the base library and the following executables. All but one mirror functionality available in the C tools. - `menuconfig `_ - `oldconfig `_ - `olddefconfig `_ - `alldefconfig `_ - `allnoconfig `_ - `allmodconfig `_ - `allyesconfig `_ - `genconfig `_ ``genconfig`` is intended to be run at build time. It generates a C header from the configuration and (optionally) information that can be used to rebuild only files that reference Kconfig symbols that have changed value. The ``menuconfig`` implementation requires Python 3. It uses ``get_wch()``, which is needed for Unicode input support. Unfortunately, ``get_wch()`` isn't available in the Python 2 version of the standard ``curses`` module. **Note:** If you install Kconfiglib with ``pip``'s ``--user`` flag, make sure that your ``PATH`` includes the directory where the executables end up. You can list the installed files with ``pip(3) show -f kconfiglib``. All releases have a corresponding tag in the git repository, e.g. ``v10.21.0``. (the latest version). `Semantic versioning `_ is used. There's been eight small changes (`1 `_, `2 `_, `3 `_, `4 `_, `5 `_, `6 `_, `7 `_, `8 `_) to the behavior of the API, which is why the major version is at 10 rather than 2. I do major version bumps for all behavior changes, even tiny ones, and most of these were fixes for baby issues in the early days of the Kconfiglib 2 API. Manual installation ~~~~~~~~~~~~~~~~~~~ Just drop ``kconfiglib.py`` and the scripts you want somewhere. There are no third-party dependencies (except for the `windows-curses `_ package on Windows, when running the terminal ``menuconfig`` implementation). Installation for the Linux kernel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ See the module docstring at the top of `kconfiglib.py `_. Getting started --------------- 1. `Install `_ the library and the utilities. Use ``pip3`` if you want to use the terminal ``menuconfig``. 2. Write `Kconfig `__ files that describe the available configuration options. 3. Generate an initial configuration with e.g. ``menuconfig`` or ``alldefconfig``. The configuration is saved as ``.config`` by default. 4. Run ``genconfig`` to generate a header file. By default, it is saved as ``config.h``. Normally, ``genconfig`` would be run automatically as part of the build. Adding new configuration output formats should be relatively straightforward. See the implementation of ``write_config()`` in `kconfiglib.py `_. The documentation for the ``Symbol.config_string`` property has some tips as well. 5. To update an old ``.config`` file after the Kconfig files have changed (e.g. to add new options), run ``oldconfig`` (prompts for values for new options) or ``olddefconfig`` (gives new options their default value). Due to Kconfig semantics, simply loading an old ``.config`` file performs an implicit ``olddefconfig``, so building will normally not be affected by having an outdated configuration. Using ``.config`` files as Make input ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``.config`` files use Make syntax and can be included directly in Makefiles to read configuration values from there. This is why ``n``-valued ``bool``/``tristate`` values are written out as ``# CONFIG_FOO is not set`` (a Make comment) in ``.config``, allowing them to be tested with ``ifdef`` in Make. If you make use of this, you might want to pass ``--config-out `` to ``genconfig`` and include the configuration file it generates instead of including ``.config`` directly. This has the advantage that the generated configuration file will always be a "full" configuration file, even if ``.config`` is outdated. Otherwise, it might be necessary to run ``old(def)config`` or ``menuconfig`` before rebuilding with an outdated ``.config``. If you use ``--sync-deps`` to generate incremental build information, you can include ``deps/auto.conf`` instead, which is also a full configuration file. Useful helper macros ~~~~~~~~~~~~~~~~~~~~ The `include/linux/kconfig.h `_ header in the Linux kernel defines some useful helper macros for testing Kconfig configuration values. ``IS_ENABLED()`` is generally useful, allowing configuration values to be tested in ``if`` statements with no runtime overhead. Incremental building ~~~~~~~~~~~~~~~~~~~~ See the docstring for ``Kconfig.sync_deps()`` in `kconfiglib.py `_ for hints on implementing incremental builds (rebuilding just source files that reference changed configuration values). Running the ``scripts/basic/fixdep.c`` tool from the kernel on the output of ``gcc -MD `` might give you an idea of how it all fits together. Library documentation --------------------- Kconfiglib comes with extensive documentation in the form of docstrings. To view it, run e.g. the following command: .. code:: sh $ pydoc(3) kconfiglib For HTML output, add ``-w``: .. code:: sh $ pydoc(3) -w kconfiglib This will also work after installing Kconfiglib with ``pip(3)``. Documentation for the ``menuconfig`` interface can be viewed in the same way: .. code:: sh $ pydoc3 menuconfig A good starting point for learning the library is to read the module docstring (which you could also just read directly at the beginning of `kconfiglib.py `_). It gives an introduction to symbol values, the menu tree, and expressions. After reading the module docstring, a good next step is to read the ``Kconfig`` class documentation, and then the documentation for the ``Symbol``, ``Choice``, and ``MenuNode`` classes. Please tell me if something is unclear or can be explained better. Library features ---------------- Kconfiglib can do the following, among other things: - **Programmatically get and set symbol values** See `allnoconfig.py `_ and `allyesconfig.py `_, which are automatically verified to produce identical output to the standard ``make allnoconfig`` and ``make allyesconfig``. - **Read and write .config and defconfig files** The generated ``.config`` and ``defconfig`` (minimal configuration) files are character-for-character identical to what the C implementation would generate (except for the header comment). The test suite relies on this, as it compares the generated files. - **Write C headers** The generated headers use the same format as ``include/generated/autoconf.h`` from the Linux kernel. - **Implement incremental builds** This uses the same scheme as the ``include/config`` directory in the kernel: Symbols are translated into files that are touched when the symbol's value changes between builds, which can be used to avoid having to do a full rebuild whenever the configuration is changed. See the ``sync_deps()`` function for more information. - **Inspect symbols** Printing a symbol or other item (which calls ``__str__()``) returns its definition in Kconfig format. This also works for symbols defined in multiple locations. A helpful ``__repr__()`` is on all objects too. All ``__str__()`` and ``__repr__()`` methods are deliberately implemented with just public APIs, so all symbol information can be fetched separately as well. - **Inspect expressions** Expressions use a simple tuple-based format that can be processed manually if needed. Expression printing and evaluation functions are provided, implemented with public APIs. - **Inspect the menu tree** The underlying menu tree is exposed, including submenus created implicitly from symbols depending on preceding symbols. This can be used e.g. to implement menuconfig-like functionality. See `menuconfig.py `_ and the minimalistic `menuconfig_example.py `_ example. Kconfig extensions ~~~~~~~~~~~~~~~~~~ The following Kconfig extensions are available: - ``source`` supports glob patterns and includes each matching file. A pattern is required to match at least one file. A separate ``osource`` statement is available for cases where it's okay for the pattern to match no files (in which case ``osource`` turns into a no-op). - A relative ``source`` statement (``rsource``) is available, where file paths are specified relative to the directory of the current Kconfig file. An ``orsource`` statement is available as well, analogous to ``osource``. - Preprocessor user functions can be defined in Python, which makes it simple to integrate information from existing Python tools into Kconfig (e.g. to have Kconfig symbols depend on hardware information stored in some other format). See the *Kconfig extensions* section in the `kconfiglib.py `_ module docstring for more information. - ``def_int``, ``def_hex``, and ``def_string`` are available in addition to ``def_bool`` and ``def_tristate``, allowing ``int``, ``hex``, and ``string`` symbols to be given a type and a default at the same time. These can be useful in projects that make use of symbols defined in multiple locations, and remove some Kconfig inconsistency. - Environment variables are expanded directly in e.g. ``source`` and ``mainmenu`` statements, meaning ``option env`` symbols are redundant. This is the standard behavior with the new `Kconfig preprocessor `_, which Kconfiglib implements. ``option env`` symbols are supported for backwards compatibility, with the caveat that they must have the same name as the environment variables they reference. A warning is printed if the names differ. - Two extra optional warnings can be enabled by setting environment variables, covering cases that are easily missed when making changes to Kconfig files: * ``KCONFIG_WARN_UNDEF``: If set to ``y``, warnings will be generated for all references to undefined symbols within Kconfig files. The only gotcha is that all hex literals must be prefixed with ``0x`` or ``0X``, to make it possible to distinguish them from symbol references. Some projects (e.g. the Linux kernel) use multiple Kconfig trees with many shared Kconfig files, leading to some safe undefined symbol references. ``KCONFIG_WARN_UNDEF`` is useful in projects that only have a single Kconfig tree though. ``KCONFIG_STRICT`` is an older alias for this environment variable, supported for backwards compatibility. * ``KCONFIG_WARN_UNDEF_ASSIGN``: If set to ``y``, warnings will be generated for all assignments to undefined symbols within ``.config`` files. By default, no such warnings are generated. This warning can also be enabled/disabled via ``Kconfig.enable/disable_undef_warnings()``. Other features -------------- - **Single-file implementation** The entire library is contained in `kconfiglib.py `_. The tools implemented on top of it are one file each. - **Runs unmodified under both Python 2 and Python 3** The code mostly uses basic Python features and has no third-party dependencies. The most advanced things used are probably ``@property`` and ``__slots__``. - **Robust and highly compatible with the standard Kconfig C tools**  The `test suite `_ automatically compares output from Kconfiglib and the C tools by diffing the generated ``.config`` files for the real kernel Kconfig and defconfig files, for all ARCHes. This currently involves comparing the output for 36 ARCHes and 498 defconfig files (or over 18000 ARCH/defconfig combinations in "obsessive" test suite mode). All tests are expected to pass. A comprehensive suite of selftests is included as well. - **Not horribly slow despite being a pure Python implementation** The `allyesconfig.py `_ script currently runs in about 1.3 seconds on the Linux kernel on a Core i7 2600K (with a warm file cache), including the ``make`` overhead from ``make scriptconfig``. Note that the Linux kernel Kconfigs are absolutely massive (over 14k symbols for x86) compared to most projects. Kconfiglib is especially speedy in cases where multiple ``.config`` files need to be processed, because the ``Kconfig`` files will only need to be parsed once. For long-running jobs, `PyPy `_ gives a big performance boost. CPython is faster for short-running jobs as PyPy needs some time to warm up. Kconfiglib also works well with the `multiprocessing `_ module. No global state is kept. - **Generates more warnings than the C implementation** Generates the same warnings as the C implementation, plus additional ones. Also detects dependency and ``source`` loops. All warnings point out the location(s) in the ``Kconfig`` files where a symbol is defined, where applicable. - **Unicode support** Unicode characters in string literals in ``Kconfig`` and ``.config`` files are correctly handled. This support mostly comes for free from Python. - **Windows support** Nothing Linux-specific is used. Universal newlines mode is used for both Python 2 and Python 3. The `Zephyr `_ project uses Kconfiglib to generate ``.config`` files and C headers on Linux as well as Windows. - **Internals that (mostly) mirror the C implementation** While being simpler to understand and tweak. Menuconfig interfaces --------------------- Two configuration interfaces are currently available: - `menuconfig.py `_ is a terminal-based configuration interface implemented using the standard Python ``curses`` module. ``xconfig`` features like showing invisible symbols and showing symbol names are included, and it's possible to jump directly to a symbol in the menu tree (even if it's currently invisible). .. image:: https://raw.githubusercontent.com/ulfalizer/Kconfiglib/screenshots/screenshots/menuconfig.gif ``menuconfig.py`` currently only supports Python 3, mostly due to ``curses.get_wch()`` not being available on Python 2. It is needed for Unicode support. There are no third-party dependencies on \*nix. On Windows, the ``curses`` modules is not available by default, but support can be added by installing the ``windows-curses`` package (which is installed automatically when Kconfiglib is installed via ``pip`` on Windows): .. code-block:: shell $ pip install windows-curses This uses wheels built from `this repository `_, which is in turn based on Christoph Gohlke's `Python Extension Packages for Windows `_. See the docstring at the top of `menuconfig.py `_ for more information about the terminal menuconfig implementation. - `RomaVis `_ has built a fully portable Python 2/3 `TkInter `_ menuconfig implementation. It is still a work-in-progress, but is already functional. See the `pymenuconfig `_ project for more information. Screenshot below: .. image:: https://raw.githubusercontent.com/RomaVis/pymenuconfig/master/screenshot.PNG While working on the terminal menuconfig implementation, I added a few APIs to Kconfiglib that turned out to be handy. ``pymenuconfig`` predates the terminal menuconfig, and so didn't have them available. Blame me for any workarounds. Examples -------- Example scripts ~~~~~~~~~~~~~~~ The `examples/ `_ directory contains some simple example scripts. Among these are the following ones. Make sure you run them with the latest version of Kconfiglib, as they might make use of newly added features. - `defconfig.py `_ has the same effect as going into ``make menuconfig`` and immediately saving and exiting. - `eval_expr.py `_ evaluates an expression in the context of a configuration. - `find_symbol.py `_ searches through expressions to find references to a symbol, also printing a "backtrace" with parents for each reference found. - `help_grep.py `_ searches for a string in all help texts. - `print_tree.py `_ prints a tree of all configuration items. - `print_config_tree.py `_ is similar to ``print_tree.py``, but dumps the tree as it would appear in ``menuconfig``, including values. This can be handy for visually diffing between ``.config`` files and different versions of ``Kconfig`` files. - `list_undefined.py `_ finds references to symbols that are not defined by any architecture in the Linux kernel. - `merge_config.py `_ merges configuration fragments to produce a complete .config, similarly to ``scripts/kconfig/merge_config.sh`` from the kernel. - `menuconfig_example.py `_ implements a configuration interface that uses notation similar to ``make menuconfig``. It's deliberately kept as simple as possible to demonstrate just the core concepts. Real-world examples ~~~~~~~~~~~~~~~~~~~ - `kconfig.py `_ from the `Zephyr `_ project handles ``.config`` and header file generation, also doing configuration fragment merging. - `genrest.py `_ generates a Kconfig symbol cross-reference, which can be viewed `here `__. - `Various utilities `_ from the `ACRN `_ project. These use the older Kconfiglib 1 API, which was clunkier and not as general (functions instead of properties, no direct access to the menu structure or properties, uglier ``__str__()`` output): - `genboardscfg.py `_ from `Das U-Boot `_ generates some sort of legacy board database by pulling information from a newly added Kconfig-based configuration system (as far as I understand it :). - `gen-manual-lists.py `_ generated listings for an appendix in the `Buildroot `_ manual. (The listing has since been removed.) - `gen_kconfig_doc.py `_ from the `esp-idf `_ project generates documentation from Kconfig files. - `SConf `_ builds an interactive configuration interface (like ``menuconfig``) on top of Kconfiglib, for use e.g. with `SCons `_. - `kconfig-diff.py `_ -- a script by `dubiousjim `_ that compares kernel configurations. - Originally, Kconfiglib was used in chapter 4 of my `master's thesis `_ to automatically generate a "minimal" kernel for a given system. Parts of it bother me a bit now, but that's how it goes with old work. Sample ``make iscriptconfig`` session ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following log should give some idea of the functionality available in the API: .. code-block:: $ make iscriptconfig A Kconfig instance 'kconf' for the architecture x86 has been created. >>> kconf # Calls Kconfig.__repr__() >>> kconf.mainmenu_text # Expanded main menu text 'Linux/x86 4.14.0-rc7 Kernel Configuration' >>> kconf.top_node # The implicit top-level menu >>> kconf.top_node.list # First child menu node >>> print(kconf.top_node.list) # Calls MenuNode.__str__() config SRCARCH string option env="SRCARCH" default "x86" >>> sym = kconf.top_node.list.next.item # Item contained in next menu node >>> print(sym) # Calls Symbol.__str__() config 64BIT bool prompt "64-bit kernel" if ARCH = "x86" default ARCH != "i386" help Say yes to build a 64-bit kernel - formerly known as x86_64 Say no to build a 32-bit kernel - formerly known as i386 >>> sym # Calls Symbol.__repr__() >>> sym.assignable # Currently assignable values (0, 1, 2 = n, m, y) (0, 2) >>> sym.set_value(0) # Set it to n True >>> sym.tri_value # Check the new value 0 >>> sym = kconf.syms["X86_MPPARSE"] # Look up symbol by name >>> print(sym) config X86_MPPARSE bool prompt "Enable MPS table" if (ACPI || SFI) && X86_LOCAL_APIC default "y" if X86_LOCAL_APIC help For old smp systems that do not have proper acpi support. Newer systems (esp with 64bit cpus) with acpi support, MADT and DSDT will override it >>> default = sym.defaults[0] # Fetch its first default >>> sym = default[1] # Fetch the default's condition (just a Symbol here) >>> print(sym) # Print it. Dependencies are propagated to properties, like in the C implementation. config X86_LOCAL_APIC bool default "y" if X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI select IRQ_DOMAIN_HIERARCHY if X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI select PCI_MSI_IRQ_DOMAIN if PCI_MSI && (X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI) >>> sym.nodes # Show the MenuNode(s) associated with it [] >>> kconfiglib.expr_str(sym.defaults[0][1]) # Print the default's condition 'X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI' >>> kconfiglib.expr_value(sym.defaults[0][1]) # Evaluate it (0 = n) 0 >>> kconf.syms["64BIT"].set_value(2) True >>> kconfiglib.expr_value(sym.defaults[0][1]) # Evaluate it again (2 = y) 2 >>> kconf.write_config("myconfig") # Save a .config >>> ^D $ cat myconfig # Generated by Kconfiglib (https://github.com/ulfalizer/Kconfiglib) CONFIG_64BIT=y CONFIG_X86_64=y CONFIG_X86=y CONFIG_INSTRUCTION_DECODER=y CONFIG_OUTPUT_FORMAT="elf64-x86-64" CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig" CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_MMU=y ... Test suite ---------- The test suite is run with .. code:: $ python(3) Kconfiglib/testsuite.py `pypy `_ works too, and is much speedier for everything except ``allnoconfig.py``/``allnoconfig_simpler.py``/``allyesconfig.py``, where it doesn't have time to warm up since the scripts are run via ``make scriptconfig``. The test suite must be run from the top-level kernel directory. It requires that the Kconfiglib git repository has been cloned into it and that the makefile patch has been applied. To get rid of warnings generated for the kernel ``Kconfig`` files, add ``2>/dev/null`` to the command to discard ``stderr``. **NOTE: Forgetting to apply the Makefile patch will cause some tests that compare generated configurations to fail** **NOTE: The test suite overwrites .config in the kernel root, so make sure to back it up.** The test suite consists of a set of selftests and a set of compatibility tests that compare configurations generated by Kconfiglib with configurations generated by the C tools, for a number of cases. See `testsuite.py `_ for the available options. The `tests/reltest `_ script runs the test suite and all the example scripts for both Python 2 and Python 3, verifying that everything works. Rarely, the output from the C tools is changed slightly (most recently due to a `change `_ I added). If you get test suite failures, try running the test suite again against the `linux-next tree `_, which has all the latest changes. I will make it clear if any non-backwards-compatible changes appear. A lot of time is spent waiting around for ``make`` and the C utilities (which need to reparse all the Kconfig files for each defconfig test). Adding some multiprocessing to the test suite would make sense too. Notes ----- * This is version 2 of Kconfiglib, which is not backwards-compatible with Kconfiglib 1. For a summary of changes between Kconfiglib 1 and Kconfiglib 2, see `kconfiglib-2-changes.txt `_. * I sometimes see people add custom output formats, which is pretty straightforward to do (see the implementations of ``write_autoconf()`` and ``write_config()`` for a template, and also the documentation of the ``Symbol.config_string`` property). If you come up with something you think might be useful to other people, I'm happy to take it in upstream. Batteries included and all that. * Kconfiglib assumes the modules symbol is ``MODULES``, which is backwards-compatible. A warning is printed by default if ``option modules`` is set on some other symbol. Let me know if you need proper ``option modules`` support. It wouldn't be that hard to add. * The test suite failures (should be the only ones) for the following Blackfin defconfigs on e.g. Linux 3.7.0-rc8 are due to  `a bug in the C implementation `_: * ``arch/blackfin/configs/CM-BF537U_defconfig`` * ``arch/blackfin/configs/BF548-EZKIT_defconfig`` * ``arch/blackfin/configs/BF527-EZKIT_defconfig`` * ``arch/blackfin/configs/BF527-EZKIT-V2_defconfig`` * ``arch/blackfin/configs/TCM-BF537_defconfig`` Thanks ------ - To `RomaVis `_, for making `pymenuconfig `_ and suggesting the ``rsource`` keyword. - To `Mitja Horvat `_, for adding support for user-defined styles to the terminal menuconfig. - To `Philip Craig `_ for adding support for the ``allnoconfig_y`` option and fixing an obscure issue with ``comment``\s inside ``choice``\s (that didn't affect correctness but made outputs differ). ``allnoconfig_y`` is used to force certain symbols to ``y`` during ``make allnoconfig`` to improve coverage. License ------- See `LICENSE.txt `_. SPDX license identifiers are used in the source code.