[FEATURE] Add integration with external programs/projects the right way

[acassis]

Is your feature request related to a problem? Please describe.

Currently there is possible to add external applications on nuttx-apps creating an "external/" directory inside apps/ [1]. However it expects that the added project has the right files inside it (Kconfig, Make.defs, Makefile, CMakelist.txt, etc).

Others RTOSes like zephyr has a similar solution (although they used a confuse name "Module" and that is a bad name since for a kernel a module means dynamic loadable component). Also in their case they expects a directory called "zephyr" inside the target project. Some examples:
https://github.com/open-quantum-safe/liboqs
https://github.com/civetweb/civetweb/tree/master/zephyr
https://github.com/lvgl/lvgl/tree/master/zephyr

This is more than a better solution, this is their way to spread their naming/awareness around all other projects in the Internet and avoids the process of creating a download script in their repository, as we do for LVGL and others.

There is a complete (and verbose) documentation about it here: https://docs.zephyrproject.org/latest/develop/modules.html

1 - It is a hidden feature, there is no Documentation about it: https://nuttx.apache.org/docs/latest/search.html?q=external&check_keywords=yes&area=default

Describe the solution you'd like

We could follow the same approach, creating a "nuttx" or "NuttX" repository inside other projects simplify the integration process.

Describe alternatives you've considered

No response

Verification

• I have verified before submitting the report.

[iamadahn]

I was just passing by, but that a great feature to improve developer experience imho. Personally I really like how it is done in zephyr, but nuttx way with makefiles and building stuff inside nuttx main repo, while developing your app in another directory which also should be located (or symlinked) inside another repo (nuttx-apps) and this ton of object files that will be everywhere, unfortunately all of that just really puts off. I know that cmake kinda solves that problems and I was able to make completely out-of-tree builds with that and nuttx cmake support is great besides documentation, but still there are a lot of boards that doesn't support it, like all esp32 stuff, so you are still tied to that old-fashioned all-in-tree way. Excuse me for that stream of consciousness, I'm just really hyped for that feature

[acassis]

@iamadahn don't worry, other comments reflect the currently reality. About the CMake you can see by the history that @simbit18 is improving it day by day, so soon all boards and apps will be supported.

About ESP32 family I think it is not complicated to add support since everything is similar to other boards (except for the HAL, that although it not the default porting way on NuttX, makes it easier to Espressif to add support to new features as chips).

[snikeguo]

#18143

[snikeguo]

everything can be “out of tree.”

Suppose both nuttx and apps have already been registered locally. Here are some typical scenarios:

Confidential SoC Features:
Imagine some registers or instructions of an MCU/SoC are confidential (for example, loongArch CPUs in China). In this case, vendors can implement the relevant CMake scripts, architecture, and other integration files out of tree, without needing to contribute confidential information to the public repository.

Private IP Blocks Without Upstream Support:
Sometimes, a particular block (IP) in the MCU is confidential, and NuttX doesn’t yet offer a driver abstraction for it. In this case, developers could implement their own drivers—such as drivers/XXX_IP and maybe architecture-specific sources like arch/loongarch/src/XXXX_IP_IMPL.c—out of tree, fully decoupled from the main tree.

Custom Boards and Applications:
As described at the beginning of this discussion, if one wants to add a new application or board, doing that out of tree is desirable for flexibility and maintainability.

If CMake could support a robust “registration” mechanism, it would make these kinds of out-of-tree workflows much easier to implement and manage. This would benefit both users with special requirements (such as confidentiality or custom hardware) and the broader community by making integration and modularization more convenient.

[acassis]

@snikeguo yes, in fact in this propose here we don't need to "register" the out-of-tree external project (called "module" in others RTOS). The idea was just to create a clone an external project inside nuttx-apps/external/ (or adding a symbolic link there) and it will verify if the project have a projectname/nuttx/ folder with the build instructions.

Maybe we can merge some ideas from #18143 but using python to register an external project should be replaced with a Makefile/CMake/C tool (since python shouldn't be required to build NuttX).

Maybe @simbit18 could have some idea how CMake can be used to do it, but we also need to replicate it in Makefile (our building system needs to support both options)

[snikeguo]

@snikeguo yes, in fact in this propose here we don't need to "register" the out-of-tree external project (called "module" in others RTOS). The idea was just to create a clone an external project inside nuttx-apps/external/ (or adding a symbolic link there) and it will verify if the project have a projectname/nuttx/ folder with the build instructions.

Maybe we can merge some ideas from #18143 but using python to register an external project should be replaced with a Makefile/CMake/C tool (since python shouldn't be required to build NuttX).

Maybe @simbit18 could have some idea how CMake can be used to do it, but we also need to replicate it in Makefile (our building system needs to support both options)

The reason for choosing a "register" mechanism is that there might be multiple NuttX repositories on the local computer. To register as a cmake package, users need to clearly specify which package they want to use.

Regarding Makefile support, I don't understand why it should be included—Zephyr, for example, does not seem to support Makefile integration at all. My bold suggestion is to remove Makefile support entirely, as it is quite outdated. If the community disagrees with dropping Makefile, we could first implement package features using cmake, and consider adding package support for Makefile gradually in the future.

[acassis]

@snikeguo because they don't use Makefile in their building system. NuttX started with Makefile and then CMake was add, but it doesn't replace CMake. In fact many people (me included) don't use CMake to build NuttX.
Zephyr started with Makefile and then replaced it with CMakefile. NuttX in the other hand added CMakefile and kept Makefile (it means more work for devs, but more options to users)

If the external project is included at apps/external/ or a symbolic link is added there then there is not need to add the register phase. It could impose some constrains, but simplify the process, everything is already there to support it. Just my two cents...