Slow performance on FlatfileClient.records.get method #44
Description
Hello everyone,
Currently we are in the process of upgrading our flatfile usage, and in doing so we've found performance issues while using the records.get method to fetch the records from a sheet.
For reference we're using the following file, which contains anonymized data:
import-template (6).csv
When running FlatfileClient.records.get(sheetId, { pageSize: 10000, pageNumber: 1, includeMessages: false }) , the method takes around 50 to 60 seconds to execute. Debugging the code we identifier that the API request (execute here) takes around 3 to 4 seconds to execute, which match our expectations. The remainder of the execution time seems to be spent on the serialization of the response (executed on parseOrThrow and subsequently on validateAndTransformRecord).
Not only the performance is slow, but when running the request on our micro-service in Kubernetes the pod is crashing with out-of-memory errors, said pod is currently provisioned with 500 MB of RAM which was previously enough to run the requests. Although we did not profile the memory usage, when we fallback to making the API request directly and returning the response of the API request, our code works as expected with the provisioned resources.
In order to check the performance we added a few console.time logs on these methods (see this commit) , created a temporary file with the following code and ran it using ts-node:
import { FlatfileClient } from "./src/index";
const token = process.env.FLATFILE_TOKEN;
const apiUrl = "https://platform.eu.flatfile.com/api/v1";
const sheetId = process.env.FLATFILE_SHEETID;
async function main() {
try {
const client = new FlatfileClient({ token, apiUrl });
await client.records.get(sheetId, {
pageSize: 10000,
pageNumber: 1,
includeMessages: false,
});
} catch (e) {
console.error(e);
}
return 0;
}
main().then(process.exit);
We wanted to check with you guys if this is the expected behavior of the library or if there is something else we need to setup in order to get a better performance in our code. As I mentioned, we're currently using the API directly with the axios library in order to make the request and return the body of the response, since in our use-case the object returned by the API is good enough to be used.