Welcome to python-monorepo-example Discussions!

[smelc]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about our monorepo design
• Share ideas of what works for you, what you tweaked
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[northshaur]

@smelc - Any chance you have an example of the python monorepo with pants? I'm beginning to head down that path and am intrigued to see what y'all may have

[GuillaumeDesforges]

Hi @northshaur, in the series of blog post (part 1 part 2) we focused on a setup without Pants, with reasons explicited in the second blog post:

We don’t demonstrate the usage of Pants for this monorepo, because Pants works better with a shared Python environment (one single requirements.txt file at the top-level of the repository), whereas our client Kaiko had business incentives to work with an environment per Python package (as described in the first post of this series). If we had been in a global sandbox scenario, Pants would be the obvious choice for preparing this monorepo for scaling, once a CI specialist has been hired for example. Pants is our favorite pick here because our monorepo is pure Python and Pants’ Python support is excellent.

Please share your experience with Pants with us as well!

[northshaur]

I completely missed that my bad! I'm aiming to setup a global sandbox right now so I'll share what I come up with !

[smelc]

Hey @northshaur, thanks for reaching out! As @GuillaumeDesforges pointed out, we didn't do Pants in the blogpost, even though we did use it for Kaiko client, which inspired us writing the blogposts.

Nevertheless our conclusion still stands nearly one year after having written the posts: if you have a global sandbox, go with Pants. My advice for the process:

1. Start with the formatting and linting tasks
2. Continue with the testing

Step 1. doesn't actually require dependencies to be super tight, because you don't run the code yet. So it's a good preparation for step 2. where things are going to become more serious.

Also final tip: try to fix the warnings that Pants outputs. In our experience, Pants' output can be very verbose when there are warnings, and the important part of the feedback gets lost in thousand of lines of silly things. Once those warnings are fixed, the meat of the errors will become way more obvious.

Let us know how it goes and happy to help if we can!

[pawanjenu]

@smelc , can we also add examples of how packaging and publishing will look like? when a new version is suposed to be released of all lib's inside the mono repo. how will it look like and stuff?

[smelc]

@pawanjenu> hello 👋

In production we had implemented packaging and publishing using Pants. But it was more or less very simple, something along (beware, untested ⚠️):

libs_roots=$(find libs -name pyproject.toml)
for dir in $libs_roots
do
  cd $(dirname $dir)
  # ... package your library in the manner you want (wheel?)
  # ... publish the artifact created when packaging
  cd -
done

If you go this scripting way, probably write the script in Python, to get support from types; because the script will grow over time.

Then my advice is to go this scripting-route for only a small amount of time. If you want to start doing fancy things, such as building/releasing libraries that have been modified by a PR in the CI, then you're better off adopting a scalable build tool like Bazel or Pants (probably Pants if your monorepo is only Python).

Scalable builds systems are designed exactly for this (see e.g. --changed-since in Pants).

[pawanjenu]

@smelc , can you also help me samples of how i can use constraints file here?

[smelc]

@pawanjenu> if you want to use a constraints file in this monorepo setup, you should probably have a single constraints.txt file at the root of the repository, and make sure that each individual library honors the constraints file, by specifying it in the script (shared along all libraries) that is used to create a single library's sandbox.

[pawanjenu]

@smelc , can you point me to some github repo which

• is monorep
• uses pants
• have some CI/CD code for publishing wheel files
• uses constraints file

any repo having any parts of this helps.
am trying to build a mono repo for my POD.
any help would be really appreciated,

[smelc]

@pawanjenu> sorry I don't know about a public repo with all these features.

[pawanjenu]

@pawanjenu> if you want to use a constraints file in this monorepo setup, you should probably have a single constraints.txt file at the root of the repository, and make sure that each individual library honors the constraints file, by specifying it in the script (shared along all libraries) that is used to create a single library's sandbox.

am. little new to this. so some sample code/folder structure or anything would help.

thanks in advance.