Skip to content

Commit 9bd8a97

Browse files
memshardedAbrilRBSczoido
authored
Feature/binary model (conan-io#3385)
* proposing section binary model * wip * wip * wip * Update reference/binary_model/custom_compatibility.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/custom_compatibility.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/custom_compatibility.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/extending.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/extending.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/extending.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/package_id.rst Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update reference/binary_model/package_id.rst * review * add links --------- Co-authored-by: Rubén Rincón Blanco <[email protected]> Co-authored-by: Carlos Zoido <[email protected]>
1 parent 2e81a8c commit 9bd8a97

File tree

13 files changed

+625
-2
lines changed

13 files changed

+625
-2
lines changed

‎images/conan_package_id.png

81.6 KB
Loading

‎images/conan_package_id_full.png

76.2 KB
Loading

‎reference.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,5 @@ Reference
1515
reference/config_files
1616
reference/extensions
1717
reference/environment
18+
reference/binary_model
1819
reference/conan_server

‎reference/binary_model.rst

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
.. _reference_binary_model:
2+
3+
The binary model
4+
================
5+
6+
This section introduces first how the ``package_id``, the package binaries identifier is computed, hashing the configuration (settings + options + dependencies versions). While the effect of ``settings`` and ``options`` is more straightforward, understanding the effects of the dependencies requires more explanations, so that will be done in its own section.
7+
8+
Conan binary model is extensible, and users can define their custom settings, options and configuration to model their own binaries characteristics.
9+
10+
Finally, the default binary compatibility model will be described, and how it can be customized to adapt to different needs.
11+
12+
13+
.. toctree::
14+
:maxdepth: 1
15+
16+
binary_model/package_id
17+
binary_model/dependencies
18+
binary_model/extending
19+
binary_model/custom_compatibility

‎reference/binary_model/custom_compatibility.rst

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
Customizing the binary compatibility
2+
====================================
3+
4+
The default binary compatibility requires an almost exact match of settings and options, and a versioned match
5+
of dependencies versions, as explained in the :ref:`previous section about dependencies<reference_binary_model_dependencies>`.
6+
7+
In summary, the required binaries ``package_id`` when installing dependencies should match by default:
8+
9+
- All the settings in the ``package_id`` except ``compiler.cppstd`` should match exactly the ones provided in the input profiles, including the compiler version. So ``compiler.version=9`` is different than ``compiler.version=9.1``.
10+
- The default behavior will assume binary compatibility among different ``compiler.cppstd`` values for C++ packages, being able to fall back to other values rather than the one specified in the input profiles, if the ``cppstd`` required by the input profile does not exist. This is controlled by the ``compatibility.py`` plugin, that can be customized by users.
11+
- All the options in the ``package_id`` should match exactly the ones provided in the input profiles.
12+
- The versions of the dependencies should match:
13+
14+
- In case of "embedding dependencies", should match the exact version, including the recipe-revision and the dependency ``package_id``. The ``package_revision`` is never included as it is assumed to be ill-formed to have more than one ``package_revision`` for the same ``package_id``.
15+
- In case of "non-embedding dependencies", the versions of the dependencies should match down to the ``minor`` version, being the ``patch``, ``recipe_revision`` and further information not taken into account.
16+
- In case of "tool dependencies", the versions of the dependencies do not affect at all by default to the consumer ``package_id``.
17+
18+
19+
These rules can be customized and changed using different approaches, depending on the needs, as explained in following sections
20+
21+
Customizing binary compatibility of settings and options
22+
--------------------------------------------------------
23+
24+
Information erasure in package_id() method
25+
++++++++++++++++++++++++++++++++++++++++++
26+
27+
Recipes can **erase** information from their ``package_id`` using their ``package_id()`` method. For example, a package containing only an executable can decide to remove the information from ``settings.compiler`` and ``settings.build_type`` from their ``package_id``, assuming that an executable built with any compiler will be valid, and that it is not necessary to store different binaries built with different compilers:
28+
29+
.. code-block:: python
30+
31+
def package_id(self):
32+
del self.info.settings.compiler
33+
del self.info.settings.build_type
34+
35+
It is also possible to assign a value for a given setting, for example if we want to have one single binary for all gcc versions included in the [>=5 <7>] range, we could do:
36+
37+
.. code-block:: python
38+
39+
def package_id(self):
40+
if self.info.settings.compiler == "gcc":
41+
version = Version(self.info.settings.compiler.version)
42+
if version >= "5.0" and version < "7.0":
43+
self.info.settings.compiler.version = "gcc5-6"
44+
45+
46+
.. note::
47+
48+
**Best practice**
49+
50+
Note that information erasure in ``package_id()`` means that 1 single ``package_id`` will represent a whole range of different settings, but the information of what exact setting was used to create the binary will be lost, and only 1 binary can be created for that range. Re-creating the package with different settings in the range, will create a new binary that overwrites the previous one (with a new package-revision).
51+
52+
If we want to be able to create, store and manage different binaries for different input settings, information erasure can't be used, and using the below ``compatibility`` approaches is recommended.
53+
54+
Read more about ``package_id()`` in:
55+
56+
- :ref:`creating_packages_configure_options_settings`
57+
- :ref:`package_id() method reference<reference_conanfile_methods_package_id>`
58+
59+
60+
The compatibility() method
61+
++++++++++++++++++++++++++
62+
63+
Recipes can define their binary compatibility rules, using their ``compatibility()`` method.
64+
For example, if we want that binaries
65+
built with gcc versions 4.8, 4.7 and 4.6 to be considered compatible with the ones compiled
66+
with 4.9 we could declare a ``compatibility()`` method like this:
67+
68+
.. code-block:: python
69+
70+
def compatibility(self):
71+
if self.settings.compiler == "gcc" and self.settings.compiler.version == "4.9":
72+
return [{"settings": [("compiler.version", v)]}
73+
for v in ("4.8", "4.7", "4.6")]
74+
75+
76+
Read more about the ``compatibility()`` method in :ref:`the compatibility() method reference<reference_conanfile_methods_compatibility>`
77+
78+
79+
The ``compatibility.py`` plugin
80+
+++++++++++++++++++++++++++++++
81+
82+
Compatibility can be defined globally via the ``compatibility.py`` plugin, in the same way that the ``compatibility()`` method does for one recipe, but for all packages globally.
83+
84+
Check the binary compatibility :ref:`compatibility.py extension <reference_extensions_binary_compatibility>`.
85+
86+
87+
88+
Customizing binary compatibility of dependencies versions
89+
---------------------------------------------------------
90+
91+
Global default package_id modes
92+
+++++++++++++++++++++++++++++++
93+
94+
The ``core.package_id:default_xxx`` configurations defined in ``global.conf`` can be used to globally change the defaults of how dependencies affect their consumers
95+
96+
.. code-block:: ini
97+
98+
core.package_id:default_build_mode: By default, 'None'
99+
core.package_id:default_embed_mode: By default, 'full_mode'
100+
core.package_id:default_non_embed_mode: By default, 'minor_mode'
101+
core.package_id:default_python_mode: By default, 'minor_mode'
102+
core.package_id:default_unknown_mode: By default, 'semver_mode'
103+
104+
.. note::
105+
106+
**Best practices**
107+
108+
It is strongly recommended that the ``core.package_id:default_xxx`` should be global, consistent and immutable accross organizations. It can be confusing to change these defaults for different projects or teams, because it will result in missing binaries.
109+
110+
It should also be consistent and shared with the consumers of generated packages if those packages are
111+
shared outside the organization, in that case sharing the ``global.conf`` file via ``conan config install``
112+
could be recommended.
113+
114+
Consider using the Conan defaults, they should be a good balance between efficiency and safety, ensuring exact re-building for embed cases, and good control via versions for non-embed cases.
115+
116+
117+
Custom package_id modes for recipe consumers
118+
++++++++++++++++++++++++++++++++++++++++++++
119+
120+
Recipes can define their default effect to their consumers, via some ``package_id_xxxx_mode`` attributes.
121+
122+
The ``package_id_embed_mode, package_id_non_embed_mode, package_id_unknown_mode`` are class attributes that can be defined in recipes to define the effect they have on their consumers ``package_id``. Can be declared as:
123+
124+
.. code-block:: python
125+
126+
from conan import ConanFile
127+
128+
class Pkg(ConanFile):
129+
...
130+
package_id_embed_mode = "full_mode"
131+
package_id_non_embed_mode = "patch_mode"
132+
package_id_unknown_mode = "minor_mode"
133+
134+
135+
Read more in :ref:`reference_conanfile_attributes_package_id_modes`
136+
137+
Custom package_id from recipe dependencies
138+
++++++++++++++++++++++++++++++++++++++++++
139+
140+
Recipes can define how their dependencies affect their ``package_id``, using the ``package_id_mode`` trait:
141+
142+
.. code-block:: python
143+
144+
from conan import ConanFile
145+
146+
class Pkg(ConanFile):
147+
def requirements(self):
148+
self.requires("mydep/1.0", package_id_mode="patch_mode")
149+
150+
151+
Using ``package_id_mode`` trait does not differentiate between the "embed" and "non-embed" cases, it is up to the user to define the correct value. It is likely that this approach should only be used for very special cases that do not have variability of shared/static libraries controlled via ``options``.
152+
153+
Note that the ``requirements()`` method is evaluated while the graph is being expanded, the dependencies do not exist yet (haven't been computed), so it is not possible to know the dependencies options.
154+
In this case it might be preferred to use the ``package_id()`` method.
155+
156+
The ``package_id()`` method can define how the dependencies affect the current package with:
157+
158+
.. code-block:: python
159+
160+
from conan import ConanFile
161+
162+
class Pkg(ConanFile):
163+
def package_id(self):
164+
self.info.requires["mydep"].major_mode()
165+
166+
The different modes that can be used are defined in :ref:`reference_conanfile_attributes_package_id_modes`

‎reference/binary_model/dependencies.rst

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
.. _reference_binary_model_dependencies:
2+
3+
The effect of dependencies on ``package_id``
4+
============================================
5+
6+
When a given package depends on a another package and uses it, the effect of dependencies can be different based on the package types:
7+
8+
For libraries:
9+
10+
- **Non-embed mode**: When an application or a shared library depends on another shared library, or when a static library depends on another static library, the "consumer" library does not do a copy of the binary artifacts of the "dependency" at all. We call it non-embed mode, the dependency binaries are not being linked or embedded in the consumer. This assumes that there are not inlined functionalities in the dependency headers, and the headers are pure interface and not implementation.
11+
- **Embed mode**: When an application or a shared library depends on a header-only or a static-library, the dependencies binaries are copied or partially copied (depending on the linker) in the consumer binary. Also when a static library depends on a header-only library, it is considered that there will be embedding in the consumer binary of such headers, as they will also contain the implementation, it is impossible that they are a pure interface.
12+
13+
For applications (``tool_requires``):
14+
15+
- **Build mode**: When some package uses a ``tool_requires`` of another package, the binary artifacts in the dependency are never copied or embedded.
16+
17+
Non-embed mode
18+
--------------
19+
20+
When we list the binaries of a package like ``openssl`` with dependencies:
21+
22+
.. code-block:: bash
23+
:emphasize-lines: 13,14
24+
25+
$ conan list openssl/3.1.2:* -r=conancenter
26+
conancenter
27+
openssl
28+
openssl/3.1.2
29+
revisions
30+
8879e931d726a8aad7f372e28470faa1 (2023-09-13 18:52:54 UTC)
31+
packages
32+
0348efdcd0e319fb58ea747bb94dbd88850d6dd1 # package_id
33+
info
34+
options
35+
shared: True
36+
...
37+
requires
38+
zlib/1.3.Z
39+
40+
This binary was a ``shared`` library, linking with ``zlib`` as a shared library.
41+
This means it was using "non-embed" mode. The default of non-embed mode is ``minor_mode``, which means:
42+
43+
- All ``zlib`` patch versions will be mapped to the same ``zlib/1.3.Z``. This means that if our ``openssl/3.1.2`` package binary ``0348efdcd0e319fb58ea747bb94dbd88850d6dd1`` binary is considered binary compatible with all ``zlib/1.3.Z`` versions (for any ``Z``), and will not require to rebuild the ``openssl`` binary.
44+
- New ``zlib`` minor versions, like ``zlib/1.4.0`` will result in a "minor-mode" identifier like ``zlib/1.4.Z``, and then, it will require a new ``openssl/3.1.2`` package binary, with a new ``package_id``
45+
46+
47+
Embed mode
48+
----------
49+
50+
The following commands illustrate the concept of embed-mode. We create a ``dep/0.1`` package with a static library, and then we create a ``app/0.1`` package with an executable that links with static library inside ``dep/0.1``. We can use the ``conan new`` command for quickly creating these two packages:
51+
52+
53+
.. code-block:: bash
54+
55+
$ mkdir dep && cd dep
56+
$ conan new cmake_lib -d name=dep -d version=0.1
57+
$ conan create . -tf=""
58+
$ cd .. && mkdir app && cd app
59+
$ conan new cmake_exe -d name=app -d version=0.1 -d requires=dep/0.1
60+
$ conan create .
61+
dep/0.1: Hello World Release!
62+
...
63+
app/0.1: Hello World Release!
64+
65+
If we now list the ``app/0.1`` binaries, we will see the binary just created:
66+
67+
68+
.. code-block:: bash
69+
:emphasize-lines: 11,12
70+
71+
$ conan list app/0.1:*
72+
Local Cache
73+
app/0.1
74+
revisions
75+
632e236936211ac2293ec33339ce582b (2023-09-25 22:34:17 UTC)
76+
packages
77+
3ca530d20914cf632eb00efbccc564da48190314
78+
info
79+
settings
80+
...
81+
requires
82+
dep/0.1#d125304fb1fb088d5b92d4f8135f4dff:9bdee485ef71c14ac5f8a657202632bdb8b4482b
83+
84+
It is now visible that the ``app/0.1`` package-id depends on the full identifier of the ``dep/0.1`` dependency, that includes both its recipe revision and ``package_id``.
85+
86+
If we do a change now to the ``dep`` code, and re-create the ``dep/0.1`` package , even if we don't bump the version, it will create a new recipe revision:
87+
88+
89+
.. code-block:: bash
90+
91+
$ cd ../dep
92+
# Change the "src/dep.cpp" code to print a new message, like "Hello Moon"
93+
$ conan create . -tf=""
94+
# New recipe revision dep/0.1#1c90e8b8306c359b103da31faeee824c
95+
96+
So if we try now to install ``app/0.1`` binary, it will fail with a "missing binary" error:
97+
98+
99+
.. code-block:: text
100+
:emphasize-lines: 7,8
101+
102+
$ conan install --requires=app/0.1
103+
ERROR: Missing binary: app/0.1:ef2b5ed33d26b35b9147c90b27b217e2c7bde2d0
104+
105+
app/0.1: WARN: Can't find a 'app/0.1' package binary 'ef2b5ed33d26b35b9147c90b27b217e2c7bde2d0' for the configuration:
106+
[settings]
107+
...
108+
[requires]
109+
dep/0.1#1c90e8b8306c359b103da31faeee824c:9bdee485ef71c14ac5f8a657202632bdb8b4482b
110+
111+
ERROR: Missing prebuilt package for 'app/0.1'
112+
113+
114+
As the ``app`` executable links with the ``dep`` static library, it needs to be rebuilt to include the latest changes, even if ``dep/0.1`` didn't bump its version, ``app/0.1`` depends on "embed-mode" on ``dep/0.1``, so it wil use down to the ``package_id`` of such dependency identifier.
115+
116+
Let's build the new ``app/0.1`` binary:
117+
118+
.. code-block:: bash
119+
:emphasize-lines: 3
120+
121+
$ cd ../app
122+
$ conan create .
123+
dep/0.1: Hello Moon Release! # Message changed to Moon
124+
...
125+
app/0.1: Hello World Release!
126+
127+
Now we will have two ``app/0.1`` different binaries:
128+
129+
.. code-block:: bash
130+
:emphasize-lines: 13,14,19,20
131+
132+
$ conan list app/0.1:*
133+
(conan2_36) λ conan list app/0.1:*
134+
Local Cache
135+
app
136+
app/0.1
137+
revisions
138+
632e236936211ac2293ec33339ce582b (2023-09-25 22:49:32 UTC)
139+
packages
140+
3ca530d20914cf632eb00efbccc564da48190314
141+
info
142+
settings
143+
...
144+
requires
145+
dep/0.1#d125304fb1fb088d5b92d4f8135f4dff:9bdee485ef71c14ac5f8a657202632bdb8b4482b
146+
ef2b5ed33d26b35b9147c90b27b217e2c7bde2d0
147+
info
148+
settings
149+
...
150+
requires
151+
dep/0.1#1c90e8b8306c359b103da31faeee824c:9bdee485ef71c14ac5f8a657202632bdb8b4482b
152+
153+
We will have these two different binaries, one of them linking with the first revision of the ``dep/0.1`` dependency (with the "Hello World" message), and the other binary with the other ``package_id`` linked with the second revision of the ``dep/0.1`` dependency (with the "Hello Moon" message).
154+
155+
The above described mode is called ``full_mode``, and it is the default for the ``embed_mode``.

0 commit comments

Comments
 (0)