feat(eval): codeguru

[nirav0999]

• integrated codeguru from the main repo
• removed dead code
• modularized eval_secure_code_gen

[gemini-code-assist]

Summary of Changes

Hello @nirav0999, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates AWS CodeGuru Security into our evaluation framework. My primary goal was to enhance our static analysis capabilities by adding a new tool for identifying security vulnerabilities. This involved creating new Python modules to orchestrate CodeGuru scans and process their results, alongside a helper script to manage the underlying AWS CLI interactions.

Highlights

• CodeGuru Integration: Implemented Python functions to programmatically interact with AWS CodeGuru Security, enabling automated vulnerability scanning of code samples.
• AWS CLI Orchestration: Developed a bash script to manage the lifecycle of CodeGuru Security scans via the AWS CLI, including uploading code artifacts, initiating scans, and retrieving findings.
• Modular Evaluation Logic: Introduced new, modular Python components for executing CodeGuru scans on multiple code samples and parsing the detailed findings into a structured format for evaluation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

Pull Request Overview

This PR integrates CodeGuru security analysis from the main repository by implementing a complete shell script and Python module for running and parsing CodeGuru security scans. The key changes include:

• Replacing TODO placeholders with fully functional CodeGuru security scanning implementation
• Adding a comprehensive shell script for AWS CodeGuru API interactions
• Creating a Python module to orchestrate scanning and parse results

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.

File	Description
eval/oracles/run_codeguru_security.sh	Replaces TODO with complete shell script for CodeGuru security scanning via AWS CLI
eval/oracles/codeguru_oracle.py	Replaces TODO with Python implementation for running CodeGuru scans and parsing results

[gemini-code-assist]

Code Review

This pull request integrates AWS CodeGuru for security analysis by adding a Python module to orchestrate scans and a shell script for AWS CLI interaction. My review focuses on enhancing the robustness and security of this new functionality. Key feedback includes resolving a potential deadlock in subprocess management, correcting a file path parsing error in the Python code, and addressing critical security vulnerabilities by removing eval calls in the shell script. I have also provided suggestions to improve argument handling and variable quoting in the shell script for better reliability.