release tcp messages from memory

[ddkohler]

• resolves protocol unpacker: old messages kept in memory #91
• resolves python 3.12 yields huge memory accumulation for yaq daemons #79
• CHANGELOG.md

[ksunden]

I would tend to prefer handling this inside of the Unpacker class, rather than reaching into private state from elsewhere.

It is not so bad considering it is all in the same package, but Unpacker is intended to be relatively self contained (and potentially spin-offable if interest was ever to accrue, though no plans to do so)

[ksunden]

Further, I would have to think harder on this to be sure that this particular clearing location would not risk missing messages

[ddkohler]

could push the job to Unpacker.feed:

    def feed(self, data: bytes):
        # Must support random access, if it does not, must be fed externally (e.g. TCP)
        pos = self._file.tell()
        self._file.seek(0, 2)
        if pos == self._file.tell():
            # read reached EOF and we are safe to clear
            self._file = io.BytesIO()
            pos = 0
        self._file.write(data)
        self._file.seek(pos)
        self.new_data.set()

This puts the rewrite inside of Unpacker and should ensure we don't miss messages. The con is that clearing the file is less scheduled and we might run into an issue where _file never gets cleared, but to me that seems unlikely in real world cases...

[ddkohler]

@ksunden lmk what you think of Unpacker's file_like argument...these changes do not respect the possibility of anything other than a BytesIO object for the Unpacker._file....

[ddkohler]

okay, using file-like methods to clear _file, rather than explicitly creating a new BytesIO object. No conflict with file_like arg for Unpacker

[untzag]

Thanks for finding this memory leak @ddkohler

@ksunden and I spent quite a while thinking about how the whole unpacker process could be better, see the story in #97

In the interest of getting this leak fixed right now, I like your solution @ddkohler. Going to merge and release this and hopefully we can come around and make a better solution e.g. #97 at a later date

[untzag]

(we did think about this implementation and decided that it's not going to cause any problems, it's purely better than what we had before)