[RHIDP-11602] [RHIDP-11600] AI Notebooks Developer Preview Backend

[JslYoon]

Hey, I just made a Pull Request!

Stories:
https://redhat.atlassian.net/browse/RHIDP-11606
https://redhat.atlassian.net/browse/RHIDP-11600
Implementing AI Notebooks backend functionalities for Session, Document, and LLM query

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or Updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)

[yangcao77]

this is very overwhelming. any existing html parser can help on this?
perhaps https://www.npmjs.com/package/htmlparser2 ?

[johnmcollier]

For reference this is the function @thepetk wrote to do the same in the techdocs mcp plugin: https://github.com/redhat-developer/rhdh-plugins/blob/main/workspaces/mcp-integrations/plugins/techdocs-mcp-extras/src/service.ts#L156-L172

[yangcao77]

I'm thinking this could block the backend for up to 30 secs.

can we make this sync file uploading process to be async?

let's say, once user upload any document, immediately return a job id

    response {
      status: 'processing',
      job_id: job_id,
      file_id: file.id,
    };

and introduce a new API to able to get the job status GET /documents/job_id/status

    response {
      status: file.status,  // 'in_progress', 'completed', 'failed'
      progress: file.bytes_processed / file.total_bytes * 100,
      error/message
    };

and rely on frontend side to monitor & display the status

[yangcao77]

or, can async handle the file, and use stream to send event for status update and frontend listen to the event. but the idea is not to use a while loop to block backend.
for example

setInterval(async () => {
      const file = await this.client.vectorStores.files.retrieve(...);

      res.write(`{
        status: file.status,
        progress: file.bytes_processed / file.total_bytes * 100,
        message/error
      })`);

      if (file.status === 'completed' || file.status === 'failed') {
        clearInterval(interval);
        res.end();
      }
    }, 1000); // every one second as you defined
  }

[yangcao77]

@karthikjeeyar which would frontend prefer?

[JslYoon]

I think instant upload & a new route to check status makes more sense, also less logical complexity @yangcao77. I would say blocking the backend is not necessary, since this file upload is async, it would not interfere with any other endpoints. Perhaps a "loading" button on the frontend while the document is uploading on frontend can help?

[JslYoon]

perhaps

response {
      status: 'processing',
      job_id: job_id,
      file_id: file.id,
    };

can be

`    response {
      status: 'processing',
      file_id: file.id,
    };`

since file_id is unique. Uploading a file with the same file_id will overwrite the file.

[yangcao77]

left a couple comments.
also I noticed there is no tests being created. please create unit tests for most of the functions. and integration tests for the endpoints.

[sonarqubecloud]

Quality Gate failed

Failed conditions
1 Security Hotspot
9.7% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[yangcao77]

cleanup?

[JslYoon]

Yep, still making some final changes!