feat(datagen): rule2code

[zhewang2001]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @zhewang2001, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've implemented a new data generation pipeline under the datagen/rule2code directory. The primary goal of this pipeline is to create and process code examples that demonstrate specific security vulnerabilities or patterns. This involves three main stages: first, scraping and structuring security rule definitions from external sources like Ruff documentation for Bandit rules; second, utilizing Large Language Models to generate Python code examples that embody these security rules; and finally, post-processing these generated examples by running static analysis on them and reformatting the results to provide detailed insights into the detected vulnerabilities and their associated security standards.

Highlights

• New Bandit Rule Scraper: I've implemented a new module, get_bandit_rules.py, which is responsible for programmatically scraping flake8-bandit rules directly from the Ruff documentation. This provides a structured and up-to-date source for security rule definitions, which can be used in subsequent data generation steps.
• LLM-based Code Generation for CodeGuru Rules: I've developed guru2code.py, a significant addition that leverages Large Language Models (LLMs) to generate Python code examples. These examples are specifically designed to demonstrate vulnerabilities detectable by CodeGuru, using existing security rule descriptions and examples as prompts. This enables the creation of a diverse dataset of vulnerable code.
• Comprehensive Post-processing and Analysis Integration: I've added post_process.py to handle the crucial steps after code generation. This module extracts the generated code, orchestrates its evaluation through static analysis (via evaluate_secure_code_gen), and then reformats the analysis results. The reformatting includes mapping findings to Common Weakness Enumerations (CWEs) and integrating information from Bandit/Ruff rules, providing a comprehensive overview of the detected issues.
• Output Path Simplification: I've made a minor but important adjustment in cwe2code.py to simplify the default output filename for generated CWE-related data, changing it from cwe2code-raw.jsonl to cwe2code.jsonl for clarity and consistency.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

Pull Request Overview

This PR introduces a complete rule2code data generation system for creating security vulnerability code examples. The implementation converts CWE and CodeGuru security rules into synthetic Python code samples that contain specific vulnerabilities, which can then be used for training or evaluating secure code generation models.

Key changes:

• Adds CWE-to-code generation pipeline with LLM-based code synthesis
• Implements CodeGuru detector rule processing and code example generation
• Creates comprehensive post-processing pipeline for vulnerability analysis and evaluation

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
datagen/rule2code/post_process.py	Implements post-processing pipeline for code analysis, vulnerability detection, and result formatting
datagen/rule2code/guru2code.py	Generates vulnerable Python code examples from CodeGuru security detector rules
datagen/rule2code/get_bandit_rules.py	Scrapes and processes Ruff/flake8-bandit security rules from documentation
datagen/rule2code/cwe2code.py	Updates output path for CWE-to-code generation

[gemini-code-assist]

Code Review

This pull request introduces a suite of scripts for generating code examples based on security rules, a feature named 'rule2code'. The new scripts handle scraping rules from documentation, generating code with language models, and post-processing the results. The implementation is a solid foundation for this new capability. My review focuses on improving the robustness, efficiency, and maintainability of these new scripts. I've identified a critical issue in the post-processing script that could corrupt Python code, suggested performance improvements for web scraping, and recommended adding type hints to enhance code quality and maintainability.