Add Gitlab Live V2 Importer by michaelehab · Pull Request #1910 · aboutcode-org/vulnerablecode

michaelehab · 2025-06-15T01:06:14Z

Solves #1903

Extract Gitlab API handling code from vulntotal gitlab datasource into a utils file.
Update vulntotal gitlab datasource to use the utils file.
Update vulntotal gitlab datasource tests to reflect the changes.
Add Gitlab Live V2 Importer.
Add Gitlab Live V2 Importer tests to test package-first mode.

TG1999 · 2025-07-01T12:18:43Z

Make changes for this as well thanks! https://github.com/aboutcode-org/vulnerablecode/blob/main/vulnerabilities/pipelines/v2_importers/gitlab_importer.py

michaelehab · 2025-07-04T09:52:18Z

@TG1999 I modified the V2 importer as well

TG1999 · 2025-07-15T11:01:48Z

vulnerabilities/pipelines/gitlab_importer.py

 from vulnerabilities.utils import build_description
 from vulnerabilities.utils import get_advisory_url
 from vulnerabilities.utils import get_cwe_id
+from vulntotal.datasources.gitlab import get_casesensitive_slug


@keshav-space what do you think is it a good idea to import vulntotal functions in vcio ? Or shall we create separate functions here ?

@michaelehab , also spdx license is missing in Gitlab vulntotal datasource. We need to have that before using it in our VCIO importers. https://github.com/aboutcode-org/vulnerablecode/blob/main/vulntotal/datasources/gitlab.py#L30

what do you think is it a good idea to import vulntotal functions in vcio ?

We can import it from VulnTotal for now. Later on we can extract these out in common utility.

@michaelehab , also spdx license is missing in Gitlab vulntotal datasource. We need to have that before using it in our VCIO importers. https://github.com/aboutcode-org/vulnerablecode/blob/main/vulntotal/datasources/gitlab.py#L30

I believe we discussed this in an earlier meeting where we said that the package-first API endpoint won't be enabled by default and users have to enable it locally which is like using vulntotal when it comes to license, that's even why I reused the vulntotal functions in VCIO's package-first mode.

* Add Gitlab Live V2 Importer * Add tests for the Gitlab Live V2 Importer * Tested functionally using the Live Evaluation API in #1969 Signed-off-by: Michael Ehab Mikhail <michael.ehab@hotmail.com>

keshav-space

Thanks @michaelehab, pipeline steps are looking good. few nits for your consideration.

keshav-space · 2025-08-19T14:59:50Z

vulnerabilities/pipelines/v2_importers/gitlab_live_importer.py

IMHO this should be inside vulnerabilities/pipelines/v2_importers/gitlab_importer.py

Based on the proposed architecture, we agreed that we'll have another parallel registry for live importers, each new live importer pipeline class will be added to this registry which the API endpoint uses to pick the relevant live importers to run.

I guess it will be better if we merge #1969 first, it adds the live evaluation API which directly utilizes the LIVE_IMPORTERS_REGISTRY.

keshav-space · 2025-08-19T15:09:33Z

vulnerabilities/pipelines/v2_importers/gitlab_live_importer.py

+def advisory_dict_to_advisory_data(
+    advisory: dict,
+    purl_type_by_gitlab_scheme,
+    gitlab_scheme_by_purl_type,
+    logger,
+    purl=None,
+    advisory_url=None,
+):
+    """
+    Convert a GitLab advisory dict to AdvisoryDataV2.
+    """


This has lots of duplicate with

vulnerablecode/vulnerabilities/pipelines/v2_importers/gitlab_importer.py

Lines 176 to 304 in dcb0511

def parse_gitlab_advisory(

file, base_path, gitlab_scheme_by_purl_type, purl_type_by_gitlab_scheme, logger

):

"""

Parse a Gitlab advisory file and return an AdvisoryData or None.

These files are YAML. There is a JSON schema documented at

https://gitlab.com/gitlab-org/advisories-community/-/blob/main/ci/schema/schema.json

Sample YAML file:

---

identifier: "GMS-2018-26"

package_slug: "packagist/amphp/http"

title: "Incorrect header injection check"

description: "amphp/http isn't properly protected against HTTP header injection."

pubdate: "2018-03-15"

affected_range: "<1.0.1"

fixed_versions:

- "v1.0.1"

urls:

- "https://github.com/amphp/http/pull/4"

cwe_ids:

- "CWE-1035"

- "CWE-937"

identifiers:

- "GMS-2018-26"

"""

with open(file) as f:

gitlab_advisory = saneyaml.load(f)

if not isinstance(gitlab_advisory, dict):

logger(

f"parse_gitlab_advisory: unknown gitlab advisory format in {file!r} with data: {gitlab_advisory!r}",

level=logging.ERROR,

)

return

# refer to schema here https://gitlab.com/gitlab-org/advisories-community/-/blob/main/ci/schema/schema.json

aliases = gitlab_advisory.get("identifiers")

advisory_id = gitlab_advisory.get("identifier")

package_slug = gitlab_advisory.get("package_slug")

advisory_id = f"{package_slug}/{advisory_id}" if package_slug else advisory_id

if advisory_id in aliases:

aliases.remove(advisory_id)

summary = build_description(gitlab_advisory.get("title"), gitlab_advisory.get("description"))

urls = gitlab_advisory.get("urls")

references = [ReferenceV2.from_url(u) for u in urls]

cwe_ids = gitlab_advisory.get("cwe_ids") or []

cwe_list = list(map(get_cwe_id, cwe_ids))

date_published = dateparser.parse(gitlab_advisory.get("pubdate"))

date_published = date_published.replace(tzinfo=pytz.UTC)

advisory_url = get_advisory_url(

file=file,

base_path=base_path,

url="https://gitlab.com/gitlab-org/advisories-community/-/blob/main/",

)

purl: PackageURL = get_purl(

package_slug=package_slug,

purl_type_by_gitlab_scheme=purl_type_by_gitlab_scheme,

logger=logger,

)

if not purl:

logger(

f"parse_yaml_file: purl is not valid: {file!r} {package_slug!r}", level=logging.ERROR

)

return AdvisoryData(

advisory_id=advisory_id,

aliases=aliases,

summary=summary,

references_v2=references,

date_published=date_published,

url=advisory_url,

original_advisory_text=json.dumps(gitlab_advisory, indent=2, ensure_ascii=False),

)

affected_version_range = None

fixed_versions = gitlab_advisory.get("fixed_versions") or []

affected_range = gitlab_advisory.get("affected_range")

gitlab_native_schemes = set(["pypi", "gem", "npm", "go", "packagist", "conan"])

vrc = RANGE_CLASS_BY_SCHEMES[purl.type]

gitlab_scheme = gitlab_scheme_by_purl_type[purl.type]

try:

if affected_range:

if gitlab_scheme in gitlab_native_schemes:

affected_version_range = from_gitlab_native(

gitlab_scheme=gitlab_scheme, string=affected_range

)

else:

affected_version_range = vrc.from_native(affected_range)

except Exception as e:

logger(

f"parse_yaml_file: affected_range is not parsable: {affected_range!r} for: {purl!s} error: {e!r}\n {traceback.format_exc()}",

level=logging.ERROR,

)

parsed_fixed_versions = []

for fixed_version in fixed_versions:

try:

fixed_version = vrc.version_class(fixed_version)

parsed_fixed_versions.append(fixed_version.string)

except Exception as e:

logger(

f"parse_yaml_file: fixed_version is not parsable`: {fixed_version!r} error: {e!r}\n {traceback.format_exc()}",

level=logging.ERROR,

)

if affected_version_range:

vrc = affected_version_range.__class__

fixed_version_range = vrc.from_versions(parsed_fixed_versions)

if not fixed_version_range and not affected_version_range:

return

affected_package = AffectedPackageV2(

package=purl,

affected_version_range=affected_version_range,

fixed_version_range=fixed_version_range,

)

return AdvisoryData(

advisory_id=advisory_id,

aliases=aliases,

summary=summary,

references_v2=references,

date_published=date_published,

affected_packages=[affected_package],

weaknesses=cwe_list,

url=advisory_url,

original_advisory_text=json.dumps(gitlab_advisory, indent=2, ensure_ascii=False),

)

we should re use the existing code and if needed we can break down parse_gitlab_advisory smaller function.

I've adressed this in my latest commit by extracting the shared logic into a common file so we can avoid duplicate code, thanks for pointing out.

…icate code #1903 Signed-off-by: Michael Ehab Mikhail <michael.ehab@hotmail.com>

michaelehab added this to GSoC 2025: On-demand live evaluation of packages and integration with VulnTotal @michaelehab Jun 15, 2025

michaelehab linked an issue Jun 15, 2025 that may be closed by this pull request

Modify the GitLab importer to support package-first mode #1903

Open

michaelehab moved this to In Progress in GSoC 2025: On-demand live evaluation of packages and integration with VulnTotal @michaelehab Jun 15, 2025

michaelehab self-assigned this Jun 16, 2025

michaelehab requested a review from keshav-space June 16, 2025 18:34

michaelehab force-pushed the 1903-gitlab-importer-package-first branch from a33f85b to c56e940 Compare July 4, 2025 15:29

TG1999 reviewed Jul 15, 2025

View reviewed changes

Add Gitlab Live V2 Importer Pipeline #1903

79429df

* Add Gitlab Live V2 Importer * Add tests for the Gitlab Live V2 Importer * Tested functionally using the Live Evaluation API in #1969 Signed-off-by: Michael Ehab Mikhail <michael.ehab@hotmail.com>

michaelehab force-pushed the 1903-gitlab-importer-package-first branch from bc7d2ea to 79429df Compare August 18, 2025 14:25

michaelehab changed the title ~~Modify Gitlab Importer to support package-first mode~~ Add Gitlab Live V2 Importer Aug 18, 2025

keshav-space requested changes Aug 19, 2025

View reviewed changes

Refactor Gitlab v2 importer pipeline and live importer to reduce dupl…

eecd1a5

…icate code #1903 Signed-off-by: Michael Ehab Mikhail <michael.ehab@hotmail.com>

michaelehab requested a review from keshav-space August 22, 2025 14:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Gitlab Live V2 Importer#1910

Add Gitlab Live V2 Importer#1910
michaelehab wants to merge 2 commits intomainfrom
1903-gitlab-importer-package-first

michaelehab commented Jun 15, 2025 •

edited

Loading

Uh oh!

TG1999 commented Jul 1, 2025

Uh oh!

michaelehab commented Jul 4, 2025

Uh oh!

TG1999 Jul 15, 2025

Uh oh!

TG1999 Jul 15, 2025 •

edited

Loading

Uh oh!

keshav-space Jul 15, 2025

Uh oh!

michaelehab Jul 17, 2025

Uh oh!

keshav-space left a comment

Uh oh!

keshav-space Aug 19, 2025

Uh oh!

michaelehab Aug 22, 2025

Uh oh!

michaelehab Aug 22, 2025

Uh oh!

keshav-space Aug 19, 2025

Uh oh!

michaelehab Aug 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	def parse_gitlab_advisory(
	file, base_path, gitlab_scheme_by_purl_type, purl_type_by_gitlab_scheme, logger
	):
	"""
	Parse a Gitlab advisory file and return an AdvisoryData or None.
	These files are YAML. There is a JSON schema documented at
	https://gitlab.com/gitlab-org/advisories-community/-/blob/main/ci/schema/schema.json

	Sample YAML file:
	---
	identifier: "GMS-2018-26"
	package_slug: "packagist/amphp/http"
	title: "Incorrect header injection check"
	description: "amphp/http isn't properly protected against HTTP header injection."
	pubdate: "2018-03-15"
	affected_range: "<1.0.1"
	fixed_versions:
	- "v1.0.1"
	urls:
	- "https://github.com/amphp/http/pull/4"
	cwe_ids:
	- "CWE-1035"
	- "CWE-937"
	identifiers:
	- "GMS-2018-26"
	"""
	with open(file) as f:
	gitlab_advisory = saneyaml.load(f)
	if not isinstance(gitlab_advisory, dict):
	logger(
	f"parse_gitlab_advisory: unknown gitlab advisory format in {file!r} with data: {gitlab_advisory!r}",
	level=logging.ERROR,
	)
	return

	# refer to schema here https://gitlab.com/gitlab-org/advisories-community/-/blob/main/ci/schema/schema.json
	aliases = gitlab_advisory.get("identifiers")
	advisory_id = gitlab_advisory.get("identifier")
	package_slug = gitlab_advisory.get("package_slug")
	advisory_id = f"{package_slug}/{advisory_id}" if package_slug else advisory_id
	if advisory_id in aliases:
	aliases.remove(advisory_id)
	summary = build_description(gitlab_advisory.get("title"), gitlab_advisory.get("description"))
	urls = gitlab_advisory.get("urls")
	references = [ReferenceV2.from_url(u) for u in urls]

	cwe_ids = gitlab_advisory.get("cwe_ids") or []
	cwe_list = list(map(get_cwe_id, cwe_ids))

	date_published = dateparser.parse(gitlab_advisory.get("pubdate"))
	date_published = date_published.replace(tzinfo=pytz.UTC)
	advisory_url = get_advisory_url(
	file=file,
	base_path=base_path,
	url="https://gitlab.com/gitlab-org/advisories-community/-/blob/main/",
	)
	purl: PackageURL = get_purl(
	package_slug=package_slug,
	purl_type_by_gitlab_scheme=purl_type_by_gitlab_scheme,
	logger=logger,
	)
	if not purl:
	logger(
	f"parse_yaml_file: purl is not valid: {file!r} {package_slug!r}", level=logging.ERROR
	)
	return AdvisoryData(
	advisory_id=advisory_id,
	aliases=aliases,
	summary=summary,
	references_v2=references,
	date_published=date_published,
	url=advisory_url,
	original_advisory_text=json.dumps(gitlab_advisory, indent=2, ensure_ascii=False),
	)
	affected_version_range = None
	fixed_versions = gitlab_advisory.get("fixed_versions") or []
	affected_range = gitlab_advisory.get("affected_range")
	gitlab_native_schemes = set(["pypi", "gem", "npm", "go", "packagist", "conan"])
	vrc = RANGE_CLASS_BY_SCHEMES[purl.type]
	gitlab_scheme = gitlab_scheme_by_purl_type[purl.type]
	try:
	if affected_range:
	if gitlab_scheme in gitlab_native_schemes:
	affected_version_range = from_gitlab_native(
	gitlab_scheme=gitlab_scheme, string=affected_range
	)
	else:
	affected_version_range = vrc.from_native(affected_range)
	except Exception as e:
	logger(
	f"parse_yaml_file: affected_range is not parsable: {affected_range!r} for: {purl!s} error: {e!r}\n {traceback.format_exc()}",
	level=logging.ERROR,
	)

	parsed_fixed_versions = []
	for fixed_version in fixed_versions:
	try:
	fixed_version = vrc.version_class(fixed_version)
	parsed_fixed_versions.append(fixed_version.string)
	except Exception as e:
	logger(
	f"parse_yaml_file: fixed_version is not parsable`: {fixed_version!r} error: {e!r}\n {traceback.format_exc()}",
	level=logging.ERROR,
	)

	if affected_version_range:
	vrc = affected_version_range.__class__

	fixed_version_range = vrc.from_versions(parsed_fixed_versions)
	if not fixed_version_range and not affected_version_range:
	return

	affected_package = AffectedPackageV2(
	package=purl,
	affected_version_range=affected_version_range,
	fixed_version_range=fixed_version_range,
	)

	return AdvisoryData(
	advisory_id=advisory_id,
	aliases=aliases,
	summary=summary,
	references_v2=references,
	date_published=date_published,
	affected_packages=[affected_package],
	weaknesses=cwe_list,
	url=advisory_url,
	original_advisory_text=json.dumps(gitlab_advisory, indent=2, ensure_ascii=False),
	)

Uh oh!

Conversation

michaelehab commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 commented Jul 1, 2025

Uh oh!

michaelehab commented Jul 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TG1999 Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michaelehab commented Jun 15, 2025 •

edited

Loading

TG1999 Jul 15, 2025 •

edited

Loading