Generic Source Provenance #2099
Conversation
joshua-zivkovic
force-pushed
the
JZ/source-provenance
branch
3 times, most recently
from
d18f672 to
6894c4f
Compare
src/buildstream/source.py
Outdated
| def load_source_provenance(self, node: MappingNode) -> None: | ||
| raise ImplError("Source plugin '{}' does not implement load_source_provenance()".format(self.get_kind())) | ||
|
|
||
| def get_source_provenance(self) -> SourceProvenance: | ||
| raise ImplError("Source plugin '{}' does not implement get_source_provenance()".format(self.get_kind())) | ||
|
|
||
| def set_source_provenance(self, source_provenance: SourceProvenance): | ||
| raise ImplError("Source plugin '{}' does not implement set_source_provenance()".format(self.get_kind())) |
There was a problem hiding this comment.
To retain backward compatibility, we can't require source plugins to implement these methods. I.e., we need (trivial) default implementations for any method that is called unconditionally by the core. API documentation will be required as well, of course.
There was a problem hiding this comment.
That's easy enough if we just go with a "do nothing" implementation
src/buildstream/source.py
Outdated
| def set_source_provenance(self, source_provenance: SourceProvenance): | ||
| raise ImplError("Source plugin '{}' does not implement set_source_provenance()".format(self.get_kind())) | ||
|
|
||
| def track(self, *, previous_sources_dir: Optional[str] = None) -> (SourceRef, SourceProvenance): |
There was a problem hiding this comment.
We need to find a way to allow returning a SourceProvenance without breaking API.
There was a problem hiding this comment.
Hmm, that might be more difficult to work out. I'll see if I can work something sensible out
There was a problem hiding this comment.
For now I've gone with combining the old and new return signatures (https://github.com/apache/buildstream/pull/2099/files#diff-48200ef302e5d841cce81834e8879a65e59db922a22605b9d7fa16b4dd5c7067R941) and added some logic to unpackage it appropriately (https://github.com/apache/buildstream/pull/2099/files#diff-48200ef302e5d841cce81834e8879a65e59db922a22605b9d7fa16b4dd5c7067R1811). I believe that's all that's required to not break the API here
joshua-zivkovic
force-pushed
the
JZ/source-provenance
branch
2 times, most recently
from
0453ec5 to
81eb24a
Compare
This allows multi-source source plugins to provide this information per source rather than as a singular top level. This is done by adding a `provenance_node` parameter to `create_source_info()` that when specified overrides the use of the source's top level source provenance info. Co-authored-by: Joshua Zivkovic <joshua.zivkovic@codethink.co.uk>
joshua-zivkovic
force-pushed
the
JZ/source-provenance
branch
2 times, most recently
from
bb44761 to
e9491c3
Compare
joshua-zivkovic
force-pushed
the
JZ/source-provenance
branch
from
e9491c3 to
51ff651
Compare
Remove harcoded SPDX attributes and make them be generic instead. Project allowed attributes are configured via the project config, these supported values a determined by buildstream-sbom's support Co-authored-by: Jürg Billeter <j@bitron.ch> Co-authored-by: Joshua Zivkovic <joshuazivkovic@codethink.co.uk>
joshua-zivkovic
force-pushed
the
JZ/source-provenance
branch
from
51ff651 to
98735d0
Compare
joshua-zivkovic
requested review from
BenjaminSchubert,
abderrahim,
cs-shadow and
gtristan
as code owners
2 participants
Towards #2069
This PR changes the approach to how buildstream handles source provenance information, moving from a hard coded implementation of attributes (which buildstream core needs to know nothing about) to a more generic, user defined approach where buildstream only needs to carry the values to the next step.
Users can now define their projects allowed set of source provenance attributes using the
source_provenance_fieldskey in the project.conf. Where attributes are used in source provenance but not defined in the project.conf, buildstream will refuse attribute and throw an error informing the user that the attribute is not present in the project.confCurrently there is no limitation on these values since neither plugins nor buildstream sbom are yet using this approach so there is no defined set of attributes plugins and users should refer to but this will soon be added to buildstream-sbom as it will depend on what buildstream-sbom supports for SPDX generation.