Convert CodeGate Pipeline to Asynchronous Processing

[lukehinds]

Background

Currently, CodeGate's security check pipeline runs synchronously, executing each step sequentially. This approach results in increased latency, slowing down the overall user experience. Given the nature of the pipeline, many steps could be executed in parallel or asynchronously to improve efficiency.

Proposed Solution

Refactor the pipeline to support an asynchronous execution model where possible. This could involve:

• Identifying independent steps that can be executed concurrently.
• Using async/await patterns where applicable.
• Leveraging task queues or event-driven mechanisms to manage execution flow.
• Ensuring appropriate error handling and logging in an asynchronous environment.

Expected Benefits

• Reduced Latency: Faster responses for users as multiple steps can run in parallel.
• Improved Scalability: The system will be better suited to handle multiple requests concurrently.
• Better Resource Utilization: Optimized execution flow reduces unnecessary wait times.

Potential Challenges

• Managing dependencies between pipeline steps.
• Handling failures and retries effectively in a non-blocking way.

Next Steps

• Assess the pipeline’s current execution model and identify independent steps.
• Determine the best approach (e.g., asyncio, worker queues, threading).
• Implement and test the asynchronous execution model.
• Monitor performance improvements and address any issues.

Additional Context

Any insights or concerns from the team regarding specific pipeline steps that require strict ordering would be helpful. Feedback on preferred async patterns or frameworks is also welcome!

[lukehinds]

It may well be worth doing #769 at the same time.

[ptelang]

We should be able to specify the pipeline as a directed acyclic graph (DAG) where each node is a step. Then the pipeline executer should execute the steps per that DAG.

[aponcedeleonch]

With the steps that we currently have the DAG could go something like this:

CodegateCli ---> CodegateSecrets ---> Rest of the steps in parallel

The reasoning is that CodegateCli returns without actually sending anything to the LLM and it doesn't take too long to execute. Then CodegateSecrets redacts any potential secrets, the output of this step can be safely used by the rest of the steps. If I'm not mistaken the rest of the steps do not depend on each other and could be executed in parallel.

[jhrozek]

Based on research that @blkt did, the biggest offender is getting the bad packages out of the RAG. The other steps are quite fast. Let's optimize the RAG step first.

Parking.