MSVS: update generation of the vcxproj embedded python script

[jcbrill]

Alternative fix for #4809 for discussion purposes.

Caution

This description no longer accurately reflects the latest commit.
An updated description is planned.

Fixes some issues with the msvs tests.

Changes:

• Replace the existing two code paths that generate the msvs tool vcxproj file with a single code path that generates the one-line python script string compatible with all cases. Prior to this change, the default test runs would not test the alternate path as SCONS_LIB_DIR is set in the test os environment.
• The msvs test case SCons version string was changed from self.scons_version to SCons.__version__. Depending on the tests run, the test framework version may not match the SCons library version being tested (e.g., retaining packaging version). The test now matches what the msvs tool uses.
• Modify the embedded vcxproj script and generated python executable string for consistency with each other with respect to SCons env and os.environ usage.
• Modify the embedded vcxproj script to retrieve SCONS_HOME and SCONS_LIB_DIR from the environment if needed.
• Priority order for msvs tool SCons library specification:
  1. env['SCONS_HOME'] literal path when generated if True
  2. os.environ.get('SCONS_HOME', <os.environ['SCONS_HOME'] path when generated>) if True
  3. os.environ.get('SCONS_LIB_DIR', <os.environ['SCONS_LIB_DIR'] path when generated>) if True
  4. if exists and is valid SCons library
  5. first in list of generated locations that exists and is valid SCons library
• Modify generation of python executable path order:
  1. env['PYTHON_ROOT'] literal path if True
  2. $(PYTHON_ROOT) if os.environ['PYTHON_ROOT'] is True
  3. sys.executable literal path
• Add bare asterisk after second argument in msvs_substitute function in testing/framework/TestSConsMSVS.py requiring keyword argument specifications for all optional arguments. The first two arguments are by position the remaining seven arguments must be keyword specified.
• The existing tests will fail if SCONS_HOME is defined in the environment in which the tests are launched and the SCONS_HOME specification does not match exactly the SCONS_LIB_DIR added to the default test environment.
• The environment that generates the msvs vcxproj file is not necessarily equivalent to the environment uses by Visual Studio when the sln file is opened (i.e., double clicked) from Windows File Explorer. This PR should minimize the effects.

Locally, passes msvc/msvs tests for all supported versions of VS using python 3.9. Passes scoop tests with SCons installed as an application and SCons installed via python.

Contributor Checklist:

• I have created a new test or updated the unit tests to cover the new/changed functionality.
• I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
• I have updated the appropriate documentation

[jcbrill]

One (hopefully unrelated) AppVeyor test failed: test\Interactive\cache-force.py.

Image: Visual Studio 2022; Environment: WINPYTHON=Python313:

571/1314 ( 43.46%) C:\Python313\python.exe test\Interactive\cache-force.py
Traceback (most recent call last):
  File "C:\projects\scons\test\Interactive\cache-force.py", line 80, in <module>
    shutil.rmtree(test.workpath('cache'))
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Python313\Lib\shutil.py", line 629, in _rmtree_unsafe
    onexc(os.unlink, fullname, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\shutil.py", line 625, in _rmtree_unsafe
    os.unlink(fullname)
    ~~~~~~~~~^^^^^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\scons\\testcmd.6716.2gdxfl31\\cache\\D9\\d9b75eba1c0a5e85a44cebd150545ffa.tmpd349df6486fb465393f0ad36bd83dff0'
Test execution time: 2.6 seconds

...

Failed the following test:
	test\Interactive\cache-force.py

NO RESULT from the following 207 tests:
...

[mwichmann]

One (hopefully unrelated) AppVeyor test failed: test\Interactive\cache-force.py.

since it only failed on one of the four appveyor runs, it seems likely to be unrelated. Let's keep an eye on it.

[bdbaddog]

Did you miss #4817 ? ;)

[jcbrill]

Did you miss #4817 ? ;)

No.

[bdbaddog]

So if I understand the impact of your changes, rather than fix the location of SCons when generating the project file, you've changed the logic so it can be impacted by user's shell env variables when run via msvs which could be different than when the project file was generated?

[jcbrill]

So if I understand the impact of your changes, rather than fix the location of SCons when generating the project file, you've changed the logic so it can be impacted by user's shell env variables when run via msvs which could be different than when the project file was generated?

Your understanding is correct.

In a nutshell, the generated vcxproj SCons library location logic is equivalent to that of the msvs tool.

Background:

• For the generating the string of the python executable which executes the one-line python script, the value of the os environment variable PYTHON_ROOT is queried. If PYTHON_ROOT is defined, the generated string is effectively $(PYTHON_ROOT) meaning the vcxproj file will use the PYTHON_ROOT environment variable.

• When generating the one-line python script, the literal value of the os environment variable values SCONS_HOME and SCONS_LIB_DIR are generated rather than generating os environment queries in the vcxproj file. Technically, the SCons environment is queried for SCONS_HOME; if undefined, the os environment is queried for SCONS_HOME; if undefined, the os environment is queried for SCONS_LIB_DIR.

  If a value was defined, the user-defined literal string is generated. If a value was not defined, a list of default values was generated. In Fixed GH Issue 4809. scoop.sh installed scons (and likely scons-local )created msvs project files would fail. #4817, only the current SCons library path is generated.

• The SCons msvs.py tool takes into account the SCons environment SCONS_HOME, os environment SCONS_HOME, and os environment SCONS_LIB_DIR. If any of these three are defined the literal value is generated; otherwise, the default list is generated.

  The testing framework script TestSConsMSVS.py only takes into account the os environment SCONS_LIB_DIR. If this is defined the literal value is generated; otherwise, the default list is generated.

  Unless I'm mistaken, SCONS_LIB_DIR is defined in the "default" test environment. This means that the "else" path is never actually tested.

  Additionally, if SCONS_HOME is defined and is not identical to the string used for SCONS_LIB_DIR (e.g., case-differences for the same path), some of the tests will fail as the SCons library uses the SCONS_HOME value and the test suite expects the SCONS_LIB_DIR value.

• There may be differences between the os environment in which the vcxproj file is generated for the first time and the os environment in which Visual Studio is launched and the solution opened.

This PR attempts to achieve:

• Unify the behavior of generating the python executable and the one-line python script.

  For the python executable:

  • If PYTHON_ROOT is defined in the SCons environment: generate the literal string path value. [new]
  • If PYTHON_ROOT is defined in the os environment: generate a reference to the environment variable. [unchanged]
  • Otherwise: generate the path for the currently running python executable. [unchanged]

  For the one-line python script:

  • If SCONS_HOME is defined in the SCons environment: generate the literal string path value. [unchanged]

  • If SCONS_HOME is defined in the os environment: generate an os environment query of SCONS_HOME with a default value of the current os environment value. If SCONS_HOME is not defined in the os environment, generate an os environment query of SCONS_HOME with a default empty string. [new]

  • If SCONS_LIB_DIR is defined in the os environment: generate an environment query of SCONS_LIB_DIR with a default value of the current os environment value. If SCONS_LIB_DIR is not defined in the os environment, generate an os environment query of SCONS_LIB_DIR with a default empty string. [new]

  • Generate checked values for the currently running SCons library path and the existing default library list with a new trailing path appended. There is a quick check to make sure that the library path exists and contains SCons (i.e., it won't add a directory to the python system path unless it contains SCons). [new]

  • The first non-empty path specification of the three "user" path definitions is used. The generated library path is used as long as it still exists when the solution is run. Otherwise, the first path specification for the default library list that contains an SCons library instance is used. [new]

    Non-empty user strings are not validated. Default library locations are validated so not to add any directories to the python system path that exist but do not contain SCons.

• There is a single path that generates the python script. The SCons msvs tool and test framework generate strings that are equal.

Here's an example used during testing (the diff format is used to show the source of the library path definition):

1. SCONS_HOME defined only in the windows terminal command-line.

   set "SCONS_HOME=C:\Work\jbrill-msvs-4809"
   python "%SCONS_HOME%\scripts\scons.py"

   With the print statement uncommented:

   exec_script_main:
     from os.path import isdir, isfile, join
     import os
     import sys
     sconslibs = lambda l: [p for p in l if p and isdir(p) and isfile(join(p, 'SCons', '__init__.py'))]
     libspec = r''
   + libspec = libspec if libspec else os.environ.get('SCONS_HOME', r'C:\Work\jbrill-msvs-4809')
     libspec = libspec if libspec else os.environ.get('SCONS_LIB_DIR', r'')
     libs = [libspec] if libspec else sconslibs([r'C:\Work\jbrill-msvs-4809'])
     libs = libs if libs else sconslibs([join(sys.prefix, 'Lib', 'site-packages', 'scons-4.10.2'), join(sys.prefix, 'scons-4.10.2'), join(sys.prefix, 'Lib', 'site-packages', 'scons'), join(sys.prefix, 'scons'), join(sys.prefix, 'Lib', 'site-packages')])
     sys.path = libs[:1] + sys.path if libs else sys.path
     print(f'libs = {libs}')
     print(f'sys.path = {sys.path}')
     import SCons.Script
     SCons.Script.main()

2. Visual Studio opened and solution loaded. Clean:

     Clean started at 9:22 AM...
     1>------ Clean started: Project: Hello, Configuration: Release Win32 ------
     1>  Starting SCons
     1>  libs = ['C:\\Work\\jbrill-msvs-4809']
     1>  sys.path = ['C:\\Work\\jbrill-msvs-4809', '', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\python314.zip', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\DLLs', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\Lib', 'C:\\Users\\jbrill\\scoop\\apps\\python\\3.14.2', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\Lib\\site-packages']
     1>  scons: Reading SConscript files ...
     1>  exec_script_main:
     1>    from os.path import isdir, isfile, join
     1>    import os
     1>    import sys
     1>    sconslibs = lambda l: [p for p in l if p and isdir(p) and isfile(join(p, 'SCons', '__init__.py'))]
     1>    libspec = r''
     1>    libspec = libspec if libspec else os.environ.get('SCONS_HOME', r'')
     1>    libspec = libspec if libspec else os.environ.get('SCONS_LIB_DIR', r'')
   + 1>    libs = [libspec] if libspec else sconslibs([r'C:\Work\jbrill-msvs-4809'])
     1>    libs = libs if libs else sconslibs([join(sys.prefix, 'Lib', 'site-packages', 'scons-4.10.2'), join(sys.prefix, 'scons-4.10.2'), join(sys.prefix, 'Lib', 'site-packages', 'scons'), join(sys.prefix, 'scons'), join(sys.prefix, 'Lib', 'site-packages')])
     1>    sys.path = libs[:1] + sys.path if libs else sys.path
     1>    print(f'libs = {libs}')
     1>    print(f'sys.path = {sys.path}')
     1>    import SCons.Script
     1>    SCons.Script.main()
     1>  scons: done reading SConscript files.
     ...
     ========== Clean: 1 succeeded, 0 failed, 0 skipped ==========
     ========== Clean completed at 9:22 AM and took 04.470 seconds ==========

3. Visual Studio opened and solution loaded. Build:

     1>------ Build started: Project: Hello, Configuration: Release Win32 ------
     1>  Starting SCons
     1>  libs = ['C:\\Work\\jbrill-msvs-4809']
     1>  sys.path = ['C:\\Work\\jbrill-msvs-4809', '', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\python314.zip', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\DLLs', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\Lib', 'C:\\Users\\jbrill\\scoop\\apps\\python\\3.14.2', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current', 'C:\\Users\\jbrill\\scoop\\apps\\python\\current\\Lib\\site-packages']
     1>  scons: Reading SConscript files ...
     1>  exec_script_main:
     1>    from os.path import isdir, isfile, join
     1>    import os
     1>    import sys
     1>    sconslibs = lambda l: [p for p in l if p and isdir(p) and isfile(join(p, 'SCons', '__init__.py'))]
     1>    libspec = r''
     1>    libspec = libspec if libspec else os.environ.get('SCONS_HOME', r'')
     1>    libspec = libspec if libspec else os.environ.get('SCONS_LIB_DIR', r'')
   + 1>    libs = [libspec] if libspec else sconslibs([r'C:\Work\jbrill-msvs-4809'])
     1>    libs = libs if libs else sconslibs([join(sys.prefix, 'Lib', 'site-packages', 'scons-4.10.2'), join(sys.prefix, 'scons-4.10.2'), join(sys.prefix, 'Lib', 'site-packages', 'scons'), join(sys.prefix, 'scons'), join(sys.prefix, 'Lib', 'site-packages')])
     1>    sys.path = libs[:1] + sys.path if libs else sys.path
     1>    print(f'libs = {libs}')
     1>    print(f'sys.path = {sys.path}')
     1>    import SCons.Script
     1>    SCons.Script.main()
     1>  scons: done reading SConscript files.
     1>  scons: Building targets ...
     ...
     ========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
     ========== Build completed at 9:22 AM and took 04.327 seconds ==========

If one wants to ALWAYS generate a path that is used, the easiest way is to define SCONS_HOME in the SCons environment as opposed to the os environment (behavior is slightly different):

import os.path
import SCons

env = Environment(
    SCONS_HOME=os.path.abspath(os.path.join(os.path.dirname(SCons.__file__), "..")),
)

sources = ['hello.cpp']
program = env.Program(
    target='hello.exe',
    source=sources,
    CPPFLAGS=['/EHsc'],
)

env.MSVSProject(
    target='Hello' + env['MSVSPROJECTSUFFIX'],
    srcs=sources,
    buildtarget=program,
    variant='Release',
)

The proposed PR is more adaptive than it was before and there is only "one" generated script instead of two.

Is it foolproof? No.

Even if this PR is not accepted, there are plenty of reasons to adapt this PR to the equivalent code in #4817 (i.e., unify both code generation paths in a single script).

I hope that answers the question and provides some background and rationale.

<whining> Comments like these take a long time to write </whining>

[bdbaddog]

@jcbrill they only take a long time when there as thorough as you usually are! :)

A few thoughts/notes:

1. I'm not sure we should use the shell environment msvs is run in to affect which scons is used by msvs to build. That's a non-trivial change from previous behavior, in theory it could require a deprecation cycle. Seems an uncommon occurrence unless the generated project file is committed and then used by another user, which seems even more likely to cause issues?
2. Currently we don't document that PYTHON_ROOT can affect SCons' behavior (see end of manpage : https://scons.org/doc/production/HTML/scons-man.html#ENVIRONMENT
3. Same is true for SCONS_HOME, it is mentioned as a Environment() variable though
4. Unifying the code which generates the snippet and modifying the test logic to allow testing when SCONS_LIB_DIR is not defined is a good update/fix.
5. I think in general we can avoid the long sys.path addition as really we only need one path to find SCons, in fact having the other paths could lead to picking up another install than the one which was used to create the msvs project file

[jcbrill]

@bdbaddog The latest commit should be functionally equivalent to #4817.

I need to do some more local testing.

There was an earlier version of the generated script that detected when the generated SCons path would not be used and printed warnings:

import os.path
import sys
scons_home = r'{scons_home}'
scons_path = r'{scons_path}'
scons_spec = scons_home if scons_home else scons_path
have_scons = bool(scons_spec and os.path.isdir(scons_spec) and os.path.isfile(os.path.join(scons_spec, 'SCons', '__init__.py')))
_ = print(f'*** SCONS WARNING: SCons was not found at the generated library path ({{scons_spec!s}}). ***') if not have_scons else None
sys.path = [scons_spec] + sys.path if have_scons else sys.path
import SCons.Script
_ = print(f'*** SCONS WARNING: SCons was found on the default python system path ({{os.path.dirname(os.path.dirname(SCons.__file__))!s}}). ***') if not have_scons else None
SCons.Script.main()

While it "appears" to work, the VS output console looked a little odd. It may have been interpreting the print statement as a command. I can't be sure.

It was simple enough to force both warnings to be printed. I'm not wild about an SCons being picked up which is not the SCons at the generated path without some kind of feedback to the user.

[jcbrill]

It appears that printing a message with the word "warning" from the embedded python script triggers different behavior.

[bdbaddog]

Why add this?

[jcbrill]

Prior to this PR (i.e., current master):

• If SCONS_HOME is set in the environment when the test is run, it has to match the SCONS_LIB_DIR string exactly.
• In msvs.py, SCONS_HOME is used before SCONS_LIB_DIR.
• In TestSConsMSVS.py, only SCONS_LIB_DIR is used.
• SCONS_LIB_DIR is added to the testenv.
• SCONS_HOME is not added to the testenv.

When forcing SCONS_LIB_DIR into the testenv, SCONS_HOME should be populated since it's used in the source code but not in the test suite.

SCONS_HOME was added to TestSConsMSVS.py for consistency with the source code.

Note: I'm testing significantly revised versions of msvs.py and TestSConsMSVS.py which I intend to push later today or in the morning tomorrow pending testing insights.

[bdbaddog]

I say that probably neither should be set by runtest.py anymore SCONS_LIB_DIR was ancient and needed prior to re-org-ing the code and the packaging.
Since it's unlikely than any user would have either set, we should be testing without them..

[jcbrill]

With regards to the current code, I believe that might require a user to set SCONS_HOME or SCONS_LIB_DIR in their environment when running the test suite with a source distribution.

For example, testing an SCons source branch with a python that does not have SCons installed. If there are any tests that actually run devenv against the generated project files would have a problem.

The scoop issue reported is effectively the same problem. A standalone SCons distribution as opposed to SCons installed as a library. SCONS_HOME is the solution with current master.

I thought at one point in time, we had to set SCONS_HOME before running the test suite but I could be wrong.

[bdbaddog]

I just commented out setting SCONS_LIB_DIR and SCONS_HOME from elif clause here and tests ran fine, with and without setting --baseline to another dir and with and without installing scons in the activated virtualenv.

diff --git a/runtest.py b/runtest.py
index ee264ecac..7cd4e9127 100755
--- a/runtest.py
+++ b/runtest.py
@@ -537,11 +537,12 @@ if scons:
     # its own modules.
     testenv['SCONS'] = scons
 elif scons_lib_dir:
+    pass
     # Because SCons is really aggressive about finding its modules,
     # it sometimes finds SCons modules elsewhere on the system.
     # This forces SCons to use the modules that are being tested.
-    testenv['SCONS_HOME'] = scons_lib_dir
-    testenv['SCONS_LIB_DIR'] = scons_lib_dir
+    # testenv['SCONS_HOME'] = scons_lib_dir
+    # testenv['SCONS_LIB_DIR'] = scons_lib_dir

 if args.scons_exec:
     testenv['SCONS_EXEC'] = '1'```

This used to be important, but hasn't been for a while as far as I'm aware.

[jcbrill]

The latest commit was modified to be more like the existing code but is subtly different. I need to write-up some of the findings, behaviors, and differences. Hopefully sooner rather than later.

The attached file below contains three versions of the msvs.py tool than can replace the msvs tool used for testing various alternatives (e.g., scoop installed scons tool, scoop installed scons python library, freestanding branches, etc).

All three versions emit diagnostic messages in the Visual Studio build window before SCons is imported and launched from the embedded script. Obviously, each file would need to be renamed to msvs.py before dropping into an SCons source tree.

The attached file scons-4809-testfiles.zip contains the following files:

• msvs_pr4818.py: a drop-in SCons/Tool/msvs.py compatible with the last commit in this PR
• msvs_pr4817.py: a drop-in SCons/Tool/msvs.py compatible with PR Fixed GH Issue 4809. scoop.sh installed scons (and likely scons-local )created msvs project files would fail. #4817
• msvs_master.py: a drop-in SCons/Tool/msvs.py compatible with the master.
• hello.cpp: test source code
• SConstruct: test SConstruct

One of the "issues" issues with the current embedded script is that it is somewhat difficult to know which SCons installation is being launched. The SCons installation being launched may not be the installation expected.

P.S.: The test/ninja/iterative_speedup.py test may have outlived it's usefulness.

[bdbaddog]

realistically I think we can just do python <path to current scons.py> where python is the python the scons being run was using.

All the rest of this like likely no longer needed?

[jcbrill]

I think that the SCons module path should only be generated for non-library installations of SCons.

Keep in mind that the generated python path can be deferred until execution of the vcxproj file by Visual Studio. As shown below, it is possible to have one python version executable importing SCons from another python version library site package location. That just seems like a bad idea.

See comment #4818 (comment) below.

[jcbrill]

Development insights.

SCons MAN Page (emphasis added):

MSVSProject()
env.MSVSProject()
Build a Microsoft Visual C++ project file and solution file.
...
For the .vcxproj file, the underlying format is the MSBuild XML Schema, and the details conform to: >https://learn.microsoft.com/en-us/cpp/build/reference/vcxproj-file-structure. The generated solution file enables Visual Studio to understand the project structure, and allows building it using MSBuild to call back to SCons. The project file encodes a toolset version that has been selected by SCons as described above. Since recent Visual Studio versions support multiple concurrent toolsets, use $MSVC_VERSION to select the desired one if it does not match the SCons default. The project file also includes entries which describe how to call SCons to build the project from within Visual Studio (or from an MSBuild command line). In some situations SCons may generate this incorrectly - notably when using the scons-local distribution, which is not installed in a way that that matches the default invocation line. If so, the $SCONS_HOME construction variable can be used to describe the right way to locate the SCons code so that it can be imported.
...

SCONS_HOME
The (optional) path to the SCons library directory, initialized from the external environment. If set, this is used to construct a shorter and more efficient search path in the $MSVSSCONS command line executed from C++ project files.

SCons/tool/msvs.py:

def getExecScriptMain(env, xml=None):
    if 'SCONS_HOME' not in env:
        env['SCONS_HOME'] = os.environ.get('SCONS_HOME')
    scons_home = env.get('SCONS_HOME')
    if not scons_home and 'SCONS_LIB_DIR' in os.environ:
        scons_home = os.environ['SCONS_LIB_DIR']
    if scons_home:
        exec_script_main = "from os.path import join; import sys; sys.path = [ r'%s' ] + sys.path; import SCons.Script; SCons.Script.main()" % scons_home
    else:
        version = SCons.__version__
        exec_script_main = "from os.path import join; import sys; sys.path = [ join(sys.prefix, 'Lib', 'site-packages', 'scons-%(version)s'), join(sys.prefix, 'scons-%(version)s'), join(sys.prefix, 'Lib', 'site-packages', 'scons'), join(sys.prefix, 'scons') ] + sys.path; import SCons.Script; SCons.Script.main()" % locals()
    if xml:
        exec_script_main = xmlify(exec_script_main)
    return exec_script_main

Code paths for the generated embedded script:

1. One or more is defined: env['SCONS_HOME'], os.environ['SCONS_HOME'], os.environ['SCONS_LIB_DIR']

   Effective embedded script:

   import sys
   sys.path = [ r'{scons_home}' ] + sys.path
   import SCons.Script
   SCons.Script.main()
   

   Possible cases:

   1. scons_home is a valid directory and contains SCons.
      SCons will be imported from the first directory in the system path.
   2. scons_home is a valid directory and does not contain SCons.
      An unnecessary directory was added to the front of the python system path which could affect imported module depending on contents.
      SCons may be found on the python system path which might be prefixed with PYTHONPATH elements.
   3. scons_home is not a valid directory
      A non-existent directory was added to the front of the python system path which has no effect.
      SCons may be found on the python system path which might be prefixed with PYTHONPATH elements.

2. None of the following are defined: env['SCONS_HOME'], os.environ['SCONS_HOME'], os.environ['SCONS_LIB_DIR']

   Effective embedded script:

   from os.path import join;
   import sys;
   sys.path = [ join(sys.prefix, 'Lib', 'site-packages', 'scons-{version}'), join(sys.prefix, 'scons-{version}'), join(sys.prefix, 'Lib', 'site-packages', 'scons'), join(sys.prefix, 'scons') ] + sys.path
   import SCons.Script
   SCons.Script.main()

   It is unlikely that all four python library directories will exist and contain SCons. It may be possible that one or more of the directories exist but do not contain SCons which could affect imported modules depending on contents.

   SCons may be found on the python system path which might be prefixed with PYTHONPATH elements.

Python location:

SCons/tool/msvs.py:

try:
    python_root = os.environ['PYTHON_ROOT']
except KeyError:
    python_executable = sys.executable
else:
    python_executable = os.path.join('$$(PYTHON_ROOT)', os.path.split(sys.executable)[1])

Code paths:

1. os.environ['PYTHON_ROOT'] is not defined
   The path of the currently running python executable is generated.
2. os.environ['PYTHON_ROOT is defined
   The value of environment variable PYTHON_ROOT is evaluated at runtime of the VS project file.

Discussion

• Issue The vcxproj file created using MSVSProject fails to build the project when SCons installed via scoop #4809 arises when SCons is installed as a standalone scoop application as opposed to as an installed library in a scoop installed python. The embedded script cannot find SCons because it is not on the python system path by default.

  Note: when SCons is installed as both a standalone tool and as a python library with scoop, the current master would find the python library installation (without warning).

  The same behavior exists when running scons.py from a source distribution (e.g., python C:\mysconsbranch\Scripts\scons.py). Without SCONS_HOME or SCONS_LIB_DIR specified, the embedded script assumes SCons can be found on the python system path. Similar to the preceding, an install SCons library version may be found and used (without warning).

  Setting SCONS_HOME in the environment works for both cases without any code changes to the current master. Setting PYTHONPATH should work as well but has not been tested. Granted, setting SCONS_HOME is somewhat inconvenient.

• Given that the evaluation of the python executable can be deferred until the VS project file is executed by Visual Studio, PR Fixed GH Issue 4809. scoop.sh installed scons (and likely scons-local )created msvs project files would fail. #4817 proposed behavior of always generating the SCons path seems like a bad idea when SCons is installed as a python library.

  For example, consider a python 3.7 virtual environment and a python 3.14 virtual environment both with SCons 4.10.1 installed. If PYTHON_ROOT is set when the vcxproj file is generated, the python executable is generated as referencing the value of PYTHON_ROOT.

  For the current master and PR MSVS: update generation of the vcxproj embedded python script #4818 without SCONS_HOME defined, SCons will be found as a library in either case.

  For PR Fixed GH Issue 4809. scoop.sh installed scons (and likely scons-local )created msvs project files would fail. #4817 without SCONS_HOME defined, running devenv from one virtual environment could refer to the library installation of the other virtual environment.

  Excerpts from a sample sequence of events showing:

  • vcxproj file built with PYTHON_ROOT set to python 3.14
  • devenv.com executed with PYTHON_ROOT set to python 3.14
  • devenv.com executed with PYTHON_ROOT set to python 3.7 and referring to a python 3.14 library location:

  VENV_SCONS=C:\Work\venv-scons-314
  PYTHON_ROOT=C:\Work\venv-scons-314
  SCONS.EXE=C:\Work\venv-scons-314\Scripts\scons.exe
  
  (venv-scons-314) C:\Work\test>"C:\Work\venv-scons-314\Scripts\scons.exe"
  scons: Reading SConscript files ...
  scons: done reading SConscript files.
  scons: Building targets ...
  cl /Fohello.obj /c hello.cpp /TP /nologo /EHsc
  hello.cpp
  link /nologo /OUT:hello.exe hello.obj
  Adding 'Hello - Release|Win32' to 'Hello.vcxproj'
  scons: done building targets.
  
  VENV_SCONS=C:\Work\venv-scons-314
  VIRTUAL_ENV=C:\Work\venv-scons-314
  PYTHON_ROOT=C:\Work\venv-scons-314\Scripts
  
  Microsoft Visual Studio Version 18.2.1.
  Copyright (C) Microsoft Corp. All rights reserved.
  Clean started at 2:26 PM...
  1>------ Clean started: Project: Hello, Configuration: Release Win32 ------
  1>  Starting SCons
  1>  proj: Using SCons path 'C:\Work\venv-scons-314\Lib\site-packages' (realpath='C:\Work\venv-scons-314\Lib\site-packages', syspath=False).
  ...
  ========== Clean: 1 succeeded, 0 failed, 0 skipped ==========
  ...
  
  Microsoft Visual Studio Version 18.2.1.
  Copyright (C) Microsoft Corp. All rights reserved.
  Build started at 2:26 PM...
  1>------ Build started: Project: Hello, Configuration: Release Win32 ------
  1>  Starting SCons
  ...
  1>  proj: Using SCons path 'C:\Work\venv-scons-314\Lib\site-packages' (realpath='C:\Work\venv-scons-314\Lib\site-packages', syspath=False).
  ...
  ========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
  ...
  
  VENV_SCONS=C:\Work\venv-scons-37
  VIRTUAL_ENV=C:\Work\venv-scons-37
  PYTHON_ROOT=C:\Work\venv-scons-37\Scripts
  
  Microsoft Visual Studio Version 18.2.1.
  Copyright (C) Microsoft Corp. All rights reserved.
  Clean started at 2:27 PM...
  1>------ Clean started: Project: Hello, Configuration: Release Win32 ------
  1>  Starting SCons
  ...
  ...                    *** |---python 3.7 exe, python 3.14 Lib-----] ***
  1>  proj: Using SCons path 'C:\Work\venv-scons-314\Lib\site-packages' (realpath='C:\Work\venv-scons-314\Lib\site-packages', syspath=False).
  ...
  ========== Clean: 1 succeeded, 0 failed, 0 skipped ==========
  
  Microsoft Visual Studio Version 18.2.1.
  Copyright (C) Microsoft Corp. All rights reserved.
  Build started at 2:27 PM...
  1>------ Build started: Project: Hello, Configuration: Release Win32 ------
  1>  Starting SCons
  ...
  ...                    *** |---python 3.7 exe, python 3.14 Lib-----] ***
  1>  proj: Using SCons path 'C:\Work\venv-scons-314\Lib\site-packages' (realpath='C:\Work\venv-scons-314\Lib\site-packages', syspath=False).
  ...
  ========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
  

• This PR unifies the three code paths: 1) scons_home defined, 2) scons_path for a non-library installation path of SCons, and 3) default library location.

  This PR differs from the existing master code in that it will only add a path to the python system path if it exists and contains SCons. In addition, the SCons import location is queried prior to import and the queried path is added to the front of the python system path if it exists and was not found earlier.

• I believe that it may be possible to defer evaluation of both the python executable and SCons module location until the vcxproj is executed by Visual Studio. I may check just for fun.

[jcbrill]

@mwichmann Another Windows permission error with a different test this time...

Visual Studio 2022; Environment: WINPYTHON=Python313:

STDERR =========================================================================
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\scons_msvc_cache.json':
  File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.7148.uuq3boni\SConstruct", line 2:
    env = Environment(tools=['msvc', 'mslink'])
  File "C:\projects\scons\SCons\Environment.py", line 1331:
    apply_tools(self, tools, toolpath)
  File "C:\projects\scons\SCons\Environment.py", line 122:
    _ = env.Tool(tool)
  File "C:\projects\scons\SCons\Environment.py", line 2197:
    tool(self)
  File "C:\projects\scons\SCons\Tool\__init__.py", line 268:
    self.generate(env, *args, **kw)
  File "C:\projects\scons\SCons\Tool\msvc.py", line 301:
    msvc_setup_env_once(env, tool=tool_name)
  File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 2269:
    msvc_setup_env(env)
  File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 2468:
    d = msvc_find_valid_batch_script(env,version)
  File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 2351:
    d = script_env(env, vc_script, args=arg)
  File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 2172:
    script_env_cache = common.read_script_env_cache()
  File "C:\projects\scons\SCons\Tool\MSCommon\common.py", line 244:
    p.unlink()
  File "C:\Python313\Lib\pathlib\_local.py", line 746:
    os.unlink(self)

That looks to be the msvc cache file attempting to be removed.

[bdbaddog]

scons.py already handles a lot of the concerns we're replicating in the generated python code inserted into the ms project file. Seems like just using scons.py will always be more correct. (and easier to maintain if additional changes in finding paths are added there).

All scoop does is run the scons.py which is in an extracted scons-local package. (For which scons.py knows how to set up the python path to run). (I tested scoop installed scons with updates from my pr)

Also note that some of the text in MSProject() manpage is really outdated and predates our current version of scons.py (and should have been updated at that time). Previously there was scons.sh and scons.bat which set up SCONS_LIB_DIR and then ran scons.py which used that to find the path. None of that is needed anymore (and hasn't been for a while).

While I think I understand your concerns in the context of what's in the manpage, and existing shell and env variables to affect this, realistically all of it should have been refactored, deprecated, and removed by now..

I don't think deferring evaluation of scons location etc until run by msvs is actually a good thing. This would allow the shell environment to impact scons's invocation in ways which violate scons' non-shell-env impact rules. (of course there are a few exceptions to this, but deferring such would allow shell env to have impacts other than the documented shell env vars).

In my competing PR #4817 I'm only referencing PYTHON_ROOT and MSVS_SCONS (variable renamed as it only impacts MSVS project generation. Dropping usage of SCONS_LIB_DIR, SCONS_HOME.
The solution in that PR is significantly simpler.
I can't find any projects on the internet besides the reporter of the scoop issue using SCONS_HOME or SCONS_LIB_DIR presently (and we're not breaking this one).

[jcbrill]

Failed checks:

• Full Test Suite on Linux / runtest (ubuntu-24.04, 3.13)
  test/ninja/iterative_speedup.py
• Test Experimental Features / experimental (macos-latest, --exclude-list=testing/ci/macos_ci_skip.txt)
  test/test/ninja/iterative_speedup.py
• Test MSVC Optional Environment / runtest-msvc (windows-latest)
  Failure to launch test? Log is odd.