Using My Utils Package with Git Submodules

This guide explains how to use the my-utils package across multiple research projects as a Git submodule, ensuring each project remains self-contained and reproducible.

Quick Reference

# Add to a new project
git submodule add https://github.com/sangmk/myutils.git myutils

# Clone a project with submodules
git clone --recursive https://github.com/yourusername/your-project.git

# Update utils to latest version
cd myutils && git pull && cd .. && git add myutils && git commit -m "Update myutils"

---

Working with Existing Projects

Cloning a Project that Uses Submodules

Method 1: Clone with submodules in one command

git clone --recursive https://github.com/yourusername/your-project.git

Method 2: Clone, then initialize submodules

git clone https://github.com/yourusername/your-project.git
cd your-project
git submodule update --init --recursive

Checking Submodule Status

# See current submodule commit
git submodule status

# Example output:
# 4a2b3c1 utils (v0.1.0)
#         ↑ commit hash that your project is locked to

---

Updating Utils

Update to Latest Version

When you want to pull recent changes from myutils:

cd myutils
git pull
cd ..

# Commit the updated reference
git add myutils
git commit -m "Update myutils to latest version"

Lock to a Specific Version

To ensure reproducibility, lock to a tagged version:

cd myutils
git fetch --tags
git checkout v0.2.0  # or any specific tag
cd ..

# Commit the version lock
git add myutils
git commit -m "Lock myutils to v0.2.0"

Check What Version You're Using

cd myutils
git describe --tags  # Shows: v0.2.0 or v0.2.0-5-g4a2b3c1

---

Making Changes to myutils

Development Workflow

When you need to fix or add functions to myutils:

# Work directly in one project's utils submodule
cd /home/workspace/project-A/myutils

# Commit and push to my-utils repo
git add .
git commit -m "Add new function for reading XYZ format"
git push origin main

# Tag new version
git tag -a v0.3.0 -m "Added XYZ reader"
git push origin v0.3.0

# Update parent project reference
cd ..
git add myutils
git commit -m "Update myutils to v0.3.0"

---

Common Workflows

Scenario 1: Starting a New Analysis Project

cd /home/workspace
mkdir new-analysis
cd new-analysis

git init
git submodule add https://github.com/sangmk/myutils.git utils

# Lock to stable version
cd utils && git checkout v0.2.0 && cd ..
git add .gitmodules utils
git commit -m "Initial commit with utils v0.2.0"

# Create analysis script
cat > analysis.py << 'EOF'
from utils.myutils.io import read_from_log
# ... your code
EOF

Scenario 2: Sharing Your Project

When someone else clones your project:

• They get the exact version of utils you used
• Your results are reproducible
• No manual setup needed

# Collaborator clones your project
git clone --recursive https://github.com/yourusername/your-project.git
cd your-project
python analysis.py  # Works immediately!

Scenario 3: Publishing a Paper

Before submission, document the exact versions:

cd your-project/utils
git describe --tags --always > ../UTILS_VERSION.txt
cd ..

# Add to README.md or paper repository
echo "Analysis uses my-utils version: $(cat UTILS_VERSION.txt)"

---

Troubleshooting

Problem: Submodule folder is empty after cloning

git submodule update --init --recursive

Problem: Changes to utils not showing up

cd myutils
git fetch origin
git pull origin main
cd ..
git add myutils
git commit -m "Update submodule reference"

Problem: Want to undo an utils update

# See history
git log -- myutils

# Reset to previous commit
git checkout <previous-commit-hash> -- myutils
git commit -m "Revert myutils to previous version"

Problem: Merge conflicts with .gitmodules

# Usually safe to accept incoming changes
git checkout --theirs .gitmodules
git add .gitmodules

---

Best Practices

1. Always tag versions in myutils before using in production analysis
2. Lock projects to specific tags for published work
3. Document which version you used in your project README
4. Test changes to utils in one project before updating others
5. Commit submodule updates separately from other changes

---

Converting Existing Projects

If you already have projects with copied utils:

cd /home/workspace/old-project

# Remove old utils copy
rm -rf utils/
git add utils/
git commit -m "Remove old utils copy"

# Add as submodule
git submodule add https://github.com/sangmk/myutils.git myutils
git add .gitmodules myutils
git commit -m "Convert myutils to submodule"

---

Additional Resources

• Git Submodules Documentation
• Pro Git Book - Chapter 7.11

---

Summary

Action	Command
Add submodule	git submodule add <url> myutils
Clone with submodules	git clone --recursive <url>
Initialize submodules	git submodule update --init --recursive
Update to latest	cd myutils && git pull && cd .. && git add myutils
Lock to version	cd myutils && git checkout v0.2.0 && cd .. && git add myutils
Check version	cd myutils && git describe --tags

Key Benefit: Each project is self-contained with versioned, traceable dependencies while avoiding code duplication.