Easiest way to integrate with the Termux repos without AutoPie's own repo?

[cryptrr]

The issues with integrating with termux repo and built packages are

• The Termux package debs has /data/data/com.myapp/files as the folder path to install in the control.tar.xz inside the deb
• Termux package source code hardcodes the full path to termux and for some packages these paths can be mission critical and hence wont work outside the termux app environment.
• Theres also this issue of modified termux pkg/apt binaries rejecting the calls to termux repos. But this is a client design and can be patched.

There are some options to bypass these like termux already does monkey patching of scripts with termux-exec

For packages where termux hardcoded paths are not critical, you can automate by extracting -> modifying -> repackaging or
preferably dynamically using tar --xform

[robertkirkman]

I do not personally believe it is technically possible to download and integrate the actual same precompiled packages from https://packages-cf.termux.dev/ without using container technology (for example proot-distro, udocker).

most of the time, when container technology is used, proot-distro is used to just install a Debian or Ubuntu aarch64 distro, which is polished and works well with apps produced by termux-generator.

But if you are familiar with proot-distro, you might know what its limitations and strengths are, here is a quick summary in case you aren't familiar yet:

strengths:

• get any Debian/Ubuntu aarch64 packages, use /usr paths normally
• compile anything that works on Debian aarch64
• does not conflict with normal Termux environment; can have both in one app seamlessly

weaknesses:

• some performance loss especially with file I/O
• sandboxed so can't easily access /storage/emulated/0 files unlike default Termux
• can't easily call Android API using termux-am or other Android-only commands
• IF using Debian (and not a nested "double termux" container), can't easily compile code with clang properly targeting android that could be used as part of another app (like building a .so file to insert into the lib/arm64-v8a folder of another app, Termux can do that when proot-distro is closed/not running, but not when it's active)

Example section

Note

(these examples are entrypoints for rootless container and container-like packages that can be installed and work in both the official "com.termux" Termux app, and also in Termux-based apps with other names generated by termux-generator, and work just the same in both because container technology is designed specifically to solve the problem of needing to change absolute paths without recompiling anything [except the container runtime, the package that implements the container support, itself])

Here is the example for proot-distro:

proot-distro install debian
proot-distro login debian

I mentioned udocker, it's also quite easy. Here is how termux-docker can be run inside udocker and install Termux packages from https://packages-cf.termux.dev/, even in a different app that's based on Termux but has any other name, as long as the package udocker was added to the bootstrap, which is quite easy:

udocker run termux/termux-docker:aarch64

That udocker example would result in the "nested com.termux Termux container inside a Termux-based app of a different name" situation.

[robertkirkman]

Now that I have posted a comment summarizing the situation and current compatibility of Termux with rootless containers,

I can explain that after considering the limitations of container technology, I decided to create termux-generator to focus on the approach of precompiling and prepackaging as many prepatched packages specifically built for the custom Termux-based app as possible, so that is the origin of termux-generator.

However, container backend packages themselves (proot-distro, udocker, and any others) are also made easier to use in custom apps by termux-generator, because "outside the container boundary", to implement the containers to begin with, so to speak, some utilities and dependencies in the app are required like curl, tar, coreutils, etc - so to add container support to any app, termux-generator can be used to help build bootstraps that already contain the necessary prepatched custom builds of the container implementation itself, and all of its dependencies necessary to download, install and run containers when the app runs.

[robertkirkman]

That all having been said,

I do think that after Termux is integrated with AutoPie, just having a few base Termux packages preinstalled in your bootstrap, plus a few larger packages to replace the equivalent ones currently available in AutoPie like clang, python, python-pip and ffmpeg,

would already make AutoPie much more powerful just on its own even without having a custom package host, because of the improved support for pip install, which in Termux, if clang is installed, can automatically compile and use many PyPi packages that have some C or C++ code in them, regardless of whether the app name is "com.termux" or not, since PyPi packages are mostly independent of platform or app name.

[cryptrr]

I created a small tool to install debs from the termux repo in AutoPie.

This rewrites the paths dynamically in the deb and installs it.

Not all packages work with this strategy. Some packages contain critical hardcoded paths to termux app directory. But for most apps these paths don't affect operation.

https://gist.github.com/cryptrr/2863c36b62f0d330f46eabcbb33d3cf6

[robertkirkman]

That's awesome! when I get a chance I will try it out!

I also have good news that we have fixed some of the errors in Termux after the update to NDK r28c, so there should no longer be an error in termux-generator currently.

It would be really cool if I can find a way to use your example to make it so termux-generator can generate APKs with custom app names that can use pkg install and pkg upgrade from the official Termux repos (even if there are some bugs).

[cryptrr]

The only friction I have seen with this strategy is the bad shebangs in the shell scripts like pip, npm etc.

Patching termux-exec to set the shebangs to #!/system/bin/env is a great low hanging fruit that fixes this issue as well as the multi user issue.

@robertkirkman Have you encountered issues with any of the packages?

[robertkirkman]

I haven't tested autopie's command functionality super heavily yet, but i can say that in my own build of my fork of autopie that i'm working on, i have tested the gui, which i have working to a mostly functional state on my phone, and it's a very beautiful interface, i really will love to use it with termux, it's just that i keep helping with a lot of other things too and i keep needing to work on termux and termux-generator to finish both of them to a more polished state. but pretty soon i hope it will be really awesome to use autopie, termux-generator, and termux all merged in a sort of full build (similar to Android AOSP!) to get the maximum convenience!

ok, so i saw some of the packages you are trying to add, and unfortunately there might be some problems at the moment.

1. it is very difficult to get libllvm / clang to build in CI in under 6 hours with enough time left to build very many other things. at first i thought it was taking less time, but when i checked again, I couldn't reproduce that lower time one run had. it might be due to fluctuations in download speed of the GitHub Actions runner.

2. Unfortunately, there is still a bug related to subpackage names in the package list that makes it difficult to add clang properly at this time - I'm going to fix it soon, then i'll tell you when it's ready here, hopefully soon. i can then also guide you if you would like, on how to build locally on your own computer so you can wait more than 6 hours and get as many packages as you want! do you have a discord account, email or are you on Termux discord server?

3. There is a genius Termux developer licy183, who techincally has invented a way to work around the GitHub Actions 6 hours CI limit, and get working artifacts after 6 hours. it technically involves starting a lot of separate builds at different times and merging them together at some point, similar to how official Termux just allows downloading packages individually. I might be able to take great inspiration from what licy183 did and make termux-generator do that, then i'd really be able to show you what the build of "retired64 termux" could be like.

"retired64 termux" is a fork of Termux in late 2024 that I compiled with a changed package name using an early prototype of termux-generator.

you can check it out if you want, in case you find an example of a prebuilt termux with a lot of packages interesting, but it's not much use since it's so old now, you should be warned that it has security vulnerabilities so is safe to use offline, but not connected to any network service, website or anything sensitive beyond simple tests in general.

https://github.com/owokitty/termux-app-retired64/releases

[cryptrr]

really awesome to use autopie, termux-generator, and termux all merged in a sort of full build (similar to Android AOSP!)

That'll be sweet. Can't wait to try it out.

it is very difficult to get libllvm / clang to build in CI in under 6 hours

I included only aarch64 to the build. So I thought it'd take less than 6 hours.

Normal minimal build takes 1 hour for aarch64 as opposed to 3 hours for all archs.

Unfortunately, there is still a bug related to subpackage names in the package list that makes it difficult to add clang properly at this time.

Ahhh. was excited to build some packages.

I just now joined the termux discord group with my current account. There's also AutoPie's discord link in the app. My email is there in the git commit details.