lindbergh: squashfs support

[udance4ever]

great work on Lindbergh Loader! 🤯

got .squashfs support working for lindbergh by implementing an overlay with write access.

1. 94GB for 19 roms compress down to 36GB freeing up 58GB (hummerxt goes from 20GB+ to 754MB - over 96%!)
2. speeds up updating gamelists as it prevents ES from recursively going down thousands of files in roms/lindbergh/*/*/

Tested on:

1. 19 Lindbergh roms
2. 7 other systems that support .squashfs: 3ds, scummvm, daphne, ps3, singe, snes-msu1, xbox

[Tovarichtch]

I don't think an overlay is the right solution. However, Sega Lindbergh ROMs really need some level of compression to avoid long gamelist loading and overall management.

[Darknior]

Ah yes really an excellent idea :)
I love Squashfs format, it same time and hard drive space :)

[udance4ever]

I don't think an overlay is the right solution.

ok fair enough, it is why I filed a request lindbergh loader execute outside the rom directory - perhaps by specifying a path to the rom on the command line as other emulators do.

However we need to consider the primary goal of the Lindbergh project is to keep existing Lindbergh Arcade cabinets running so we can't expect this to be implemented short-term.

It was a relatively simple change to conditionally create an overlay in /var/run/squashfs without getting too much in the way of existing code.

Are there other issues you see I am not aware of?

[bryanforbes]

This may need to be refactored as part of Generator (and a refactor of when the generator is created) because it's rare for decisions to be made on the system/emulator/core outside of Generator and its subclasses.

Before talking about this, we need to figure out if an overlay is even the right option.

[dmanlfc]

why use overlay?
using the /var/run/squashfs/ as rw is not an ideal location for preservation of any rom updates. why here?

[udance4ever]

I'm open to other ideas & saw this was the most straightforward to implement and get working.

The bottleneck is lindbergh doesn't run outside (eg. /usr/bin/lindbergh and it's shared .so libraries) and the config file in system/configs/lindbergh is only a template so it requires the rom directory to be writeable so the generator can copy all these files into it at runtime.

The solution squashfs already provides is the use of an overlay - this is what it was designed for as far as I can tell.

The changes are only temporary to allow the rom to boot. All changes can be safely discarded after the game exits because the config file is generated every time.

I also noticed the code can be reused as other squashfs mounts like daphne and singe want to write a .dat file into the directory if you don't remember to generate it before mksquashfs. Use of the overlay logic allows the title to boot instead of crapping out because it doesn't have write access as the .dat file can be regenerated (until the rom is upgraded to include it). This is a separate discussion but thought I'd add additional rationale.

[udance4ever]

just heard from a dev in lindbergh.

lindbergh-loader/lindbergh-loader#72 (comment)

it sounds like we should be able to run the binary out of /usr/bin, create a config file in ~/configs (using -c) & leave the NVRAM & EPROM files in saves

I have to assume this has already been tried.

Anyone able to share what the sticking issue was & why we copy binaries, shared libraries & config file into the rom directory?

When I get a moment, Ill tinker with it in a different branch & if I am able to get lindbergh to boot without copying any files, I will make a separate PR so squashfs support is a separate task.

[Tovarichtch]

This?

https://github.com/lindbergh-loader/lindbergh-loader/blob/efe34307e38cd9a9e84680d125fbf9371a72ab6c/src/lindbergh/lindbergh.c#L285

It would still need the correct libs in it, but at least config files of the loader would be somewhere in /userdata/configs/lindbergh

[udance4ever]

yes - that was the option we were pointed to.

new options

I just found the SRAM and EEPROM options buried in the code:

https://github.com/lindbergh-loader/lindbergh-loader/blob/efe34307e38cd9a9e84680d125fbf9371a72ab6c/src/lindbergh/config.c#L1110-L1114

extra libraries

where would be a good place to copy extra libraries to? would ~/configs/lindbergh/extralibs be okay?

then we could add ~/configs/lindbergh/extralibs to LD_LIBRARY_PATH

legacy libraries

the last wrench in the equation is code that deletes a legacy libsegaapi.so inside the rom directory

so if this is indeed necessary (as some squashfs files may contain it), then we should probably be safe and just mount the squashfs as writable anyways. (just my 2 cents)

batocera.linux/package/batocera/core/batocera-configgen/configgen/configgen/generators/lindbergh/lindberghGenerator.py

Lines 862 to 864 in 7ab1bd1

	# remove any legacy libsegaapi.so
	if Path(romDir, "libsegaapi.so").exists():
	Path(romDir, "libsegaapi.so").unlink()

[dmanlfc]

Previously when I added the emulator the option didn't work. We can retry again.

[udance4ever]

Previously when I added the emulator the option didn't work. We can retry again.

good to know you've tried! You might be right, I notice it accepts the command line option but still seems to create a new eeprom.bin file inside the rom directory (so it's unclear what the purpose of the local one is as it is a different length file)

The shared libraries appears to be the bigger issue.

I added /usr/bin/lindbergh to LD_LIBRARY_PATH and it still throws an error - is it statically linked to . by chance? Are we able to update the build to dynamically link libraries if this is the case?

I propose we move forward with the .squashfs implementation. Happy to refactor Generator sooner than later if this is important.

The implementation as is works around all the above issues with minimal changes.

I will resolve merge conflicts and retest when I get the green light.

[aderumier]

Hi, can't we have some kind of writetable overlay like wsquashfs for windows ? (maybe made it generic, it could help for mugen too for example)

[Darknior]

Hi, can't we have some kind of writetable overlay like wsquashfs for windows ? (maybe made it generic, it could help for mugen too for example)

For sure ... so many files in Linberg games ... it is a pain to transfer or delete by FTP :(
A single file system should help a lot ... but like it is not native, i dont know if Squashfs is possible ...