Optical Character Recognition (OCR)

[tolik518]

Recognizing factorials directly in a image could be a fun addition and would give the bot a much wider coverage. As far as I have seen most OCR crates are fairly easy to integrate.

Downsides:

• slower response times/latency (since currently we're working on the comments sequentially)
• higher resource use with OCR
• false positives/negatives
  • might make the bot look dumb

Crates

I don't have a strong preference right now, but from what I have seen floneum looks promising as it uses models which could work best with structured inputs.
Traditional OCR like tesseract would probably be a lot less resource hungry, but less accurate and would probably need more configuration. Tesseract biggest upside by far is that it should be more resource-efficient than "model-based" OCRs

[Aras14HD]

Sounds like something one would enable only on very few subs (only the factorial/termial ones) and/or with commands. Enabling it for all images would be very expensive (cpu wise). I would also only do it for image posts, not images in comments or linked images (imgur).

One thing I would also look at is cross posts, it would be nice to support them too (including also plain text).

On what ocr to use, I lean resource cheap, but would also want to estimate or test actual load.

[tolik518]

and/or with commands.

I'm afraid, if the OCR functionality is kept behind a command, it won't be used that much. Enabling it for all images would be fine with me (I think?), but you made a good point with linked images - linked images could take a lot of time if the host is slow.
As long as the OCR functionality isn't async i.e running in a different service or in a thread, it wouldn't be feasible I think.

One thing I would also look at is cross posts, it would be nice to support them too (including also plain text).

Makes sense, I like it. I don't think I saw one example yet though were the factorial wasn't part of the title in the new post as well :D

On what ocr to use, I lean resource cheap, but would also want to estimate or test actual load.

Same, but I have 0 clue so far how resource heavy the options are. I have only experience with an OCR api, but this isn't an option for me in the case of factorion.

[Aras14HD]

leptess would be a tesseract based option. But still what to use depends, does the server have gpu? Do we want to primarily recognize text in screenshots or do we need to get it in pictures as well?

And even tesseract will still take a few seconds. So it might make sense to have an ocr thread anyway:

1. Images with partial Comments are sent to it
2. It extracts the text, maybe translates the markup,
3. appends it to the comment text
4. and sends the filled comment back
5. After getting comments (maybe in the method), the main thread recieves them and adds them to comments)

[tolik518]

The server has sadly no GPU, the setup is the following

Intel Core i7-7700
HDD: 2x SSD M.2 NVMe 512 GB
RAM: 2x RAM 32768 MB DDR4

Text in screenshots as well as in pictures would be awesome of course, but I'm not sure how feasible that would be. Pure text/plain screenshots would be much easier to work with of course and the certainty for a positive response would be higher.

Currently I'm even thinking about maybe having a separate ocr service that runs in its own container (or same container, but own process) that gets called factorion of the comments/posts have any media_urls (or however it's called). This could happen roughly between get_comments and extract. So we would simply "enrich" the comments with the ocr data.
This is not far away from what you have wrote I think. I don't have any hard views on the implementation, but we should make sure that OCR would be fairly non-blocking and optional