Resources rebuilt unncessarily during install or on subsequent builds. #95

sergey-239 · 2022-09-20T05:24:36Z

Dear Joseph,
for some reason, the install target triggers rebuild of build/resources.os, which in turn makes the default target out of date.
Could you please change this behavior, so when there is no actual need to rebuild the default target (it was just successfully built), it is not linked again by 'scons install' invocation.

jcoffland · 2022-09-21T12:47:05Z

Does my recent commit fix this?

sergey-239 · 2022-09-21T15:50:14Z

Joseph,

unfortunately, no. Building of build/resources.os is still performed in install target, so the default targets get rebuilt.
Thank you,
Sergey

sergey-239 · 2022-09-22T02:10:25Z

Joseph.
after some digging into the problem I found that your patch is actually prevents the recreation of generated files. Though I would use more explicit test for install target like 'install' in BUILD_TARGETS. Nevertheless scons decides for some reason that the build/resources.os deps are changed and compiles it.
The solution is to invoke scons install with --implicit-deps-unchanged option switch: then it does not compile/build anything even without this commit.
For the record I am using python3-scons-4.4.0-2.fc38.noarch

jcoffland · 2022-09-23T12:33:55Z

I'm not sure why --implicit-deps-unchanged is necessary for your build. It must have something to do with your particular configuration. When I run scons install. No files are rebuilt. The default build is a static build, maybe that has something to do with it.

sergey-239 · 2022-09-23T12:43:47Z

Joseph,
I don't know either. I have a) very basic knowledge of python and b) I need to study the scons source to get to the roots of the problem. Unfortunately I have no time for this now. Presently I am building shared library, but I was encountering the same effect when I was building camotics with static cbang during install build of the camotics. Now I have a workaround, Probably, I would look into this issue some day.
Regards,
Sergey

sergey-239 · 2022-09-23T20:03:06Z

Joseph, I have the hypotesy.
During default target build, scons prints out the following from resource tool emitter:
('build/resources.cpp', 'src/resources.data/data2.cpp')
During install target build, scons outputs this:
('build/resources.cpp', 'build/resources.data/data2.cpp')
This essentially means that the the nodes graph is different between two scons invocations, and scons has no cached time/checksum for node build/resources.data/data2.cpp as a dependency of 'build/resources.cpp', so it marks the latter as need to rebuild node.

Then, the next thing (just after the default target build):

ls -l --time-style=+%s build/resources.cpp src/resources.data/data2.cpp build/resources.data/data2.cpp
ls: cannot access 'src/resources.data/data2.cpp': No such file or directory
-rw-r--r--. 1 mockbuild mock  686 1663962660 build/resources.cpp
-rw-r--r--. 1 mockbuild mock 4469 1663962660 build/resources.data/data2.cpp

this explains everything, but the fact that you do not observe the similar behavior.

Unfortunately, I am not capable to fix this properly in a reasonable time: despite of my thirty five years experience in sw engineering, the functional languages are above my procedural way of thinking...

Regards,
Sergey

Edit:
for your reference, I use two commands:
scons staticlib=0 sharedlib=1 soname=libcbang0.so.1.7.0 with_openssl=0 v8_compress_pointers=0 cxxstd=c++17 debug=1 strict=0 libsuffix=64 'disable_local=zlib bzip2 lz4 sqlite3 expat boost libevent re2 libyaml'
and
scons install prefix=%{buildroot}%{_prefix} staticlib=0 sharedlib=1 soname=libcbang0.so.1.7.0 with_openssl=0 v8_compress_pointers=0 cxxstd=c++17 debug=1 strict=0 libsuffix=64 'disable_local=zlib bzip2 lz4 sqlite3 expat boost libevent re2 libyaml'
(I stripped out the compiler and linker flags for clarity)

jcoffland · 2022-09-23T21:09:19Z

Yeah src/resources.data/data2.cpp does not and should not exist. I'm not sure how that got in there. When I build with your command line it prints this:

('build/resources.cpp', 'build/resources.data/data2.cpp', 'build/resources.data/data3.cpp', 'build/resources.data/data4.cpp', 'build/resources.data/data5.cpp', 'build/resources.data/data6.cpp', 'build/resources.data/data7.cpp', 'build/resources.data/data8.cpp', 'build/resources.data/data9.cpp', 'build/resources.data/data10.cpp', 'build/resources.data/data11.cpp')

Did you remove the license files from src/resources/licenses/? Maybe I should add an option to just disable the compiled in resources. The only reason they are there in cbang is to be able to display these license files.

sergey-239 · 2022-09-23T21:21:20Z

Ah, yes, I forgot to mention that any third-party code is removed in the %prep phase, so there is only two subdirectories under src/: cbang and resources. also, all licenses from /resources/licensed removed except cbang.txt. This is to ensure that no third-party code is used in build
Yeah src/resources.data/data2.cpp does not and should not exist
It does not exist, see the output of ls command. The emitter returns this result for some reason.

sergey-239 · 2022-09-23T22:26:20Z

Maybe I should add an option to just disable the compiled in resources. The only reason they are there in cbang is to be able to display these license files.

Yes, I think that if lib source is disabled by disable_local, then appropriate license should not be included as well. The reason to include license texts at all is obvious.

sergey-239 · 2022-09-24T13:04:08Z

Two more things to start the install target build using commands I posted above:

You need to create a ./lib/libcbang0.so symlink to /.lib/cbang0.so.1.7.0
You need to export CBANG_LIBNAME=cbang0
to let scons find the built library

sergey-239 · 2022-09-25T01:15:08Z

Joseph,

#99 does not fix the issue with wrong deps list yet:
default target:
('build/resources.cpp', 'src/resources.data/data2.cpp', 'src/resources.data/data3.cpp')
install target:
('build/resources.cpp', 'build/resources.data/data2.cpp', 'build/resources.data/data3.cpp')

Regards,
Sergey

sergey-239 · 2022-09-25T04:18:23Z

Joseph,

The problem's source is identified.
Unless the build/resources.data/data2.cpp exists, the File('%s/data%d.cpp' % (data_dir, id)) at config/resources/init: returns the object with path modified to src/....
You do not observe the same behavior as you most likely start the default build from dirty source tree while mock does this always from scratch.
To simulate this behavior just remove build/resources.data and scons cache file before invoking scons.
Also, there is some strange thing with build/resources.data/data2.cpp's dependencies:
a) with or without my patch #99 and some(all) disabled libs, the deps are all files existing in the src/resources/ subtree;
b) the deps are listed three times contrary to regular source files in src/ subtree:

  +-build/resources.data/data2.os
  | +-build/resources.data/data2.cpp
  | | +-src/resources
  | | | +-src/resources/licenses
  | | | | +-src/resources/licenses/boost.txt
  | | | | +-src/resources/licenses/bzip2.txt
  | | | | +-src/resources/licenses/cbang.txt
  | | | | +-src/resources/licenses/expat.txt
  | | | | +-src/resources/licenses/libevent.txt
  | | | | +-src/resources/licenses/libyaml.txt
  | | | | +-src/resources/licenses/lz4.txt
  | | | | +-src/resources/licenses/re2.txt
  | | | | +-src/resources/licenses/sqlite.txt
  | | | | +-src/resources/licenses/test.backup.file~
  | | | | +-src/resources/licenses/zlib.txt
  | | | +-src/resources/test.backup.file~
  | | +-src/cbang/util/Resource.h
  | | +-src/resources/licenses
  | | | +-src/resources/licenses/boost.txt
  | | | +-src/resources/licenses/bzip2.txt
  | | | +-src/resources/licenses/cbang.txt
  | | | +-src/resources/licenses/expat.txt
  | | | +-src/resources/licenses/libevent.txt
  | | | +-src/resources/licenses/libyaml.txt
  | | | +-src/resources/licenses/lz4.txt
  | | | +-src/resources/licenses/re2.txt
  | | | +-src/resources/licenses/sqlite.txt
  | | | +-src/resources/licenses/test.backup.file~
  | | | +-src/resources/licenses/zlib.txt
  | | +-src/resources/test.backup.file~
  | | +-src/resources/licenses/boost.txt
  | | +-src/resources/licenses/bzip2.txt
  | | +-src/resources/licenses/cbang.txt
  | | +-src/resources/licenses/expat.txt
  | | +-src/resources/licenses/libevent.txt
  | | +-src/resources/licenses/libyaml.txt
  | | +-src/resources/licenses/lz4.txt
  | | +-src/resources/licenses/re2.txt
  | | +-src/resources/licenses/sqlite.txt
  | | +-src/resources/licenses/test.backup.file~
  | | +-src/resources/licenses/zlib.txt
  | +-src/cbang/util/Resource.h

One more confirmation (from scons --debug=explain install):

('build/resources.cpp', 'build/resources.data/data2.cpp')
scons: rebuilding `build/resources.os' because:
           `src/cbang/util/Resource.h' is no longer a dependency
           `src/cbang/Exception.h' is no longer a dependency
           `src/cbang/FileLocation.h' is no longer a dependency
           `src/cbang/SmartPointer.h' is no longer a dependency
           `src/cbang/Throw.h' is no longer a dependency
           `src/cbang/debug/StackTrace.h' is no longer a dependency
           `src/cbang/StdTypes.h' is no longer a dependency
           `src/cbang/RefCounter.h' is no longer a dependency
           `src/cbang/SStream.h' is no longer a dependency
           `src/cbang/debug/StackFrame.h' is no longer a dependency
           `src/cbang/Deallocators.h' is no longer a dependency
           `src/cbang/util/FormatCheck.h' is no longer a dependency

Regards,
Sergey

sergey-239 · 2022-09-26T03:34:47Z

Joseph,

it looks like the list of dependencies includes whatever is in the supplied source directory as you use simple directory scanner in defining resources builder. I understand that this does not influences outcome, the worst it could do is just trigger unnecessary re-builds when something get touched in the source directory, that is not my case anyway.
As I mentioned already, I have the workaround so this is not an urgent thing to fix.
At least this issue enforced me to start study scons: I have never seen it before. To be honest, I like scons more and more and as a side effect I will improve my python skills by exploring scons ;)

Regards,
Sergey

jcoffland · 2022-09-26T10:43:06Z

This seems to be the exact same problem: https://jira.mongodb.org/browse/SERVER-33111 It looks like they fixed it by patching scons but the fix was apparently not applied upstream.

Edit1: Actually, it was applied upstream: SCons/scons#2811 I'm still having what appears to be the same problem with scons v4.0.1
Edit2: The above patch has been applied in v4.0.1 but the problem persists. This appears to be a variant of the original issue.

jcoffland · 2022-09-26T12:31:04Z

I've created a simplified test case which exhibits the bug.

scons-v4.0.1-generated-file-rebuild-on-install-bug-20220926.zip

Running scons I get:

scons: Reading SConscript files ...
New targets ('build/resources.cpp', 'src/resources.data/hello.txt.cpp')
scons: done reading SConscript files.
scons: Building targets ...
build(["build/resources.cpp", "build/resources.data/hello.txt.cpp"], ["src/resources"])
Writing build/resources.cpp
Writing build/resources.data/hello.txt.cpp
g++ -o build/resources.data/hello.txt.o -c -Isrc build/resources.data/hello.txt.cpp
g++ -o build/resources.o -c -Isrc build/resources.cpp
g++ -o test build/resources.o build/resources.data/hello.txt.o
Install file: "test" as "tmp/test"
scons: done building targets.

Then run scons --debug=explain install:

scons: Reading SConscript files ...
New targets ('build/resources.cpp', 'build/resources.data/hello.txt.cpp')
scons: done reading SConscript files.
scons: Building targets ...
scons: rebuilding `build/resources.o' because `src/dep/dep.h' is no longer a dependency
g++ -o build/resources.o -c -Isrc build/resources.cpp
scons: `install' is up to date.
scons: done building targets.

It rebuilds every time I switch from scons install to scons and back. Repeated runs of the same target do not rebuild.

It appears that running scons vs scons install writes the dependencies to the scons database differently.

sergey-239 · 2022-09-27T10:51:17Z

Joseph
thank you for the references.

It appears that running scons vs scons install writes the dependencies to the scons database differently.

No, not only in this situation. Try this:

rm -rf build/resources.*
scons --debug=explain
scons --debug=explain
scons --debug=explain

On first invocation of scons it rebuilds resources as there are no cpp files
On second - because of scons bug with dependencies
And only on third it says that target is up to date. However, it rebuilds build_info as marked 'always build' but then this does not make default target out of date as checksum of build_info.os remains the same, obviously.

By the way, I completed the #99. Now it does not include unnecessary licenses and also builds proper dependencies lists, so the mod time check inside resources_build no longer needed.
As for scons bug with deps, I will continue to find if this behavior might be avoided without patching scons

Regards,
Sergey

jcoffland · 2022-09-27T11:00:23Z

It will rebuild on each invocation like this too:

rm -rf build/resources.*
scons
scons
scons  install
scons

Re: Install target question #95

sergey-239 · 2022-09-27T11:10:18Z

It will rebuild on each invocation like this too:
rm -rf build/resources.*
scons
scons
scons  install
scons

Yes, I did not check all possible combinations :)
I checked the scons source installed on my system form Fedora repo: it has at least some changes from SCons/scons#3340 incorporated.
Regards,
Sergey

jcoffland · 2022-09-27T11:15:39Z

My scons also has that "fix" applied. It seems that scons has more problems with tracking dependencies correctly when a builder adds additional targets. Thanks for helping track this down. It would be great if we could find a solution that does not require modifying scons but you work around is ok for your situation and the extra rebuilding does not bother me too much so it's probably not a high priority issue.

jcoffland · 2022-09-27T11:18:12Z

I've reopened this issue to track the bug even though the true bug is upstream in scons.

jcoffland added a commit that referenced this issue Sep 21, 2022

Avoid building anything during install, re: #95

a712b75

sergey-239 closed this as completed Sep 22, 2022

sergey-239 mentioned this issue Sep 25, 2022

Re: Install target question #95 #99

Merged

jcoffland added a commit that referenced this issue Sep 27, 2022

Merge pull request #99 from sergey-239/fix_resources_inclusion

fe6e272

Re: Install target question #95

jcoffland changed the title ~~Install target question~~ Resources rebuilt unncessarily during install or on subsequent builds. Sep 27, 2022

jcoffland added the bug label Sep 27, 2022

jcoffland reopened this Sep 27, 2022

jcoffland added the build system label Sep 27, 2022

jcoffland removed the bug label Dec 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resources rebuilt unncessarily during install or on subsequent builds. #95

Resources rebuilt unncessarily during install or on subsequent builds. #95

sergey-239 commented Sep 20, 2022

jcoffland commented Sep 21, 2022

sergey-239 commented Sep 21, 2022

sergey-239 commented Sep 22, 2022 •

edited

Loading

jcoffland commented Sep 23, 2022

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 23, 2022 •

edited

Loading

jcoffland commented Sep 23, 2022

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 24, 2022

sergey-239 commented Sep 25, 2022 •

edited

Loading

sergey-239 commented Sep 25, 2022 •

edited

Loading

sergey-239 commented Sep 26, 2022 •

edited

Loading

jcoffland commented Sep 26, 2022 •

edited

Loading

jcoffland commented Sep 26, 2022

sergey-239 commented Sep 27, 2022

jcoffland commented Sep 27, 2022

sergey-239 commented Sep 27, 2022

jcoffland commented Sep 27, 2022

jcoffland commented Sep 27, 2022 •

edited

Loading

Resources rebuilt unncessarily during install or on subsequent builds. #95

Resources rebuilt unncessarily during install or on subsequent builds. #95

Comments

sergey-239 commented Sep 20, 2022

jcoffland commented Sep 21, 2022

sergey-239 commented Sep 21, 2022

sergey-239 commented Sep 22, 2022 • edited Loading

jcoffland commented Sep 23, 2022

sergey-239 commented Sep 23, 2022 • edited Loading

sergey-239 commented Sep 23, 2022 • edited Loading

jcoffland commented Sep 23, 2022

sergey-239 commented Sep 23, 2022 • edited Loading

sergey-239 commented Sep 23, 2022 • edited Loading

sergey-239 commented Sep 24, 2022

sergey-239 commented Sep 25, 2022 • edited Loading

sergey-239 commented Sep 25, 2022 • edited Loading

sergey-239 commented Sep 26, 2022 • edited Loading

jcoffland commented Sep 26, 2022 • edited Loading

jcoffland commented Sep 26, 2022

sergey-239 commented Sep 27, 2022

jcoffland commented Sep 27, 2022

sergey-239 commented Sep 27, 2022

jcoffland commented Sep 27, 2022

jcoffland commented Sep 27, 2022 • edited Loading

sergey-239 commented Sep 22, 2022 •

edited

Loading

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 23, 2022 •

edited

Loading

sergey-239 commented Sep 25, 2022 •

edited

Loading

sergey-239 commented Sep 25, 2022 •

edited

Loading

sergey-239 commented Sep 26, 2022 •

edited

Loading

jcoffland commented Sep 26, 2022 •

edited

Loading

jcoffland commented Sep 27, 2022 •

edited

Loading