Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .githooks/pre-commit
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env sh
# Same check as fluid-pub/actions go-workload-ci "Verify formatting".
set -e

repo_root="$(git rev-parse --show-toplevel 2>/dev/null)" || {
echo "pre-commit: not inside a Git work tree" >&2
exit 1
}

cd "$repo_root" || exit 1

if ! command -v gofmt >/dev/null 2>&1; then
echo "pre-commit: gofmt not found in PATH" >&2
exit 1
fi

unformatted="$(gofmt -l .)"
if [ -n "$unformatted" ]; then
echo "pre-commit: gofmt would reformat these files (run: gofmt -w .):" >&2
echo "$unformatted" >&2
exit 1
fi
44 changes: 44 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
version: 2
updates:
- package-ecosystem: gomod
directory: "/"
schedule:
interval: weekly
day: monday
target-branch: develop
open-pull-requests-limit: 5
commit-message:
prefix: chore(deps)
labels:
- dependencies
- go

- package-ecosystem: docker
directory: "/"
schedule:
interval: weekly
day: monday
target-branch: develop
open-pull-requests-limit: 5
commit-message:
prefix: chore(deps)
labels:
- dependencies
- docker

- package-ecosystem: github-actions
directory: "/"
schedule:
interval: weekly
day: monday
target-branch: develop
open-pull-requests-limit: 5
commit-message:
prefix: chore(deps)
labels:
- dependencies
- github-actions
groups:
github-actions:
patterns:
- "*"
20 changes: 20 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: CI

on:
pull_request:
push:
branches:
- main
- develop

permissions:
contents: read

jobs:
ci:
uses: fluid-pub/actions/.github/workflows/go-workload-ci.yml@v1
with:
workload_kind: probe
core_repository: fluid-pub/probe-core
core_ref: develop
go_version: "1.25"
53 changes: 53 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: CodeQL

on:
push:
branches:
- develop
- main
pull_request:
branches:
- develop
- main
schedule:
- cron: "27 3 * * 1"

permissions:
actions: read
contents: read
security-events: write

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6

- name: Checkout shared core
uses: actions/checkout@v6
with:
repository: fluid-pub/probe-core
ref: develop
path: core

- name: Align go.mod replace for CI layout
run: |
go mod edit -replace fluid/probes/core=./core
go mod download

- uses: actions/setup-go@v6
with:
go-version: "1.25"
cache-dependency-path: go.sum

- name: Initialize CodeQL
uses: github/codeql-action/init@v4
with:
languages: go

- name: Autobuild
uses: github/codeql-action/autobuild@v4

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
20 changes: 20 additions & 0 deletions .github/workflows/release-on-semver-tag.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Release on semver tag

on:
push:
tags:
- "[0-9]+.[0-9]+.[0-9]+"
- "[0-9]+.[0-9]+.[0-9]+-*"

permissions:
contents: write
packages: write

jobs:
release:
uses: fluid-pub/actions/.github/workflows/go-workload-release.yml@v1
with:
workload_kind: probe
binary_name: gitlab-probe
core_repository: fluid-pub/probe-core
core_ref: develop
65 changes: 65 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, built with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Dependency directories (remove the comment below to include it)
# vendor/

# Go workspace file
go.work

# Build directory
build/
gitlab-probe
gitlab-agent

# State directory
state/

# Data directory (repositories clone, etc.)
data/

# IDE files
.vscode/
.idea/
*.swp
*.swo
*~

# OS generated files
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db

# Logs
*.log

# Environment variables
.env
.env.local
env.secrets

# Local probe config (use probe.example.yml as template)
config/probe.yml

# Coverage reports
coverage.out
coverage.html

# Temporary files
*.tmp
*.temp

6 changes: 6 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Required when this module is the root of fluid-pub/probe-gitlab on GitHub.
# Monorepo dev can use replace => ../core in go.mod without initializing this submodule.
[submodule "core"]
path = core
url = https://github.com/fluid-pub/probe-core.git
branch = develop
125 changes: 125 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
---
name: gitlab-probe
description: Expert Go developer for the Fluid GitLab probe (inventory, repository sync, RAG code_files)
---

You are an expert Go developer working on the **Fluid GitLab probe** (`fluid-pub/probe-gitlab`): a **probe** (not an execution agent) that collects GitLab data and pushes entities to the Fluid control plane over HTTP (`/probes`).

## Persona

- Integrate with **GitLab API v4** and optional **local repository mirrors** for file indexing.
- Compose **`fluid/probes/core`** for lifecycle, state files, control plane HTTP, schema push, and `runtime_config` merge.
- Write idiomatic Go: explicit errors, English log messages, no customer-specific URLs or names in versioned files.

## Tech stack

- **Go 1.25+** (see `go.mod`; CI uses Go 1.25 for go-git dependencies).
- **`fluid/probes/core`** via submodule `core/` (or `replace ../core` in monorepo dev).
- **YAML v3** for config and local state snapshots.
- **go-git** (`github.com/go-git/go-git/v5`) for clone/fetch/checkout — **never** `exec.Command("git", ...)`.
- **Distroless** runtime image: static binary only, **no `git` binary** in the container.

## Repository layout

| Path | Role |
|------|------|
| `core/` | Submodule → `fluid-pub/probe-core` |
| `cmd/main.go` | Entrypoint, signals, optional `MergedConfigProvider` |
| `cmd/version.go` | Semver aligned with release tag |
| `internal/probe/probe.go` | Composes `*core.Probe`, registers entities |
| `internal/probe/entities/` | Per-entity `Refresh` / `Save` (users, groups, projects, repositories, code_files) |
| `internal/manager/` | Thin wrapper over `core/state.Manager` |
| `internal/gitlab/` | GitLab REST client |
| `internal/repositories/` | go-git sync + filesystem scan for `code_files` |
| `internal/config/` | YAML load, validation, `source_view` URL helpers |
| `internal/models/` | Entity structs |
| `config/probe.example.yml` | Generic template (versioned) |
| `config/schema.yml` | Entity contract (embedded in Docker image) |
| `config/probe.yml` | Local config (gitignored) |
| `env.secrets.example` | `GITLAB_TOKEN`, `FLUID_CONTROLPLANE_*` |
| `.github/workflows/` | CI / release via `fluid-pub/actions` |

Gitignored at runtime: `state/`, `data/repositories/`, `env.secrets`.

## Architecture

1. **`main`** loads `config/probe.yml`, wraps config in `core.MergedConfigProvider` when control plane is configured.
2. **`probe.NewProbe`** builds GitLab client, `manager.NewManager`, `core.NewProbe`, registers entities listed in `data.entities`.
3. Each entity runs on its own **`refresh_interval`** (goroutine per entity in probe-core).
4. **`SaveEntity`** persists YAML under `state.dir` and enqueues ingest to the control plane when connected.
5. **`repositories` / `code_files`**: go-git shallow clone under `data.repositories.base_dir`, then walk files (respect `index_files`, RAG on `fields.content.rag`).

Entities: `users`, `groups`, `projects`, `repositories`, `code_files` — see `config/schema.yml`.

## Configuration (current shape)

```yaml
probe:
name: gitlab
version: "0.1.0"
gitlab:
url: "https://gitlab.example.com"
token: "${GITLAB_TOKEN}"
api_version: "v4"
group_id: "your-group-path"
data:
include_subgroups: true
entities:
- name: users
refresh_interval: "15m"
# ... groups, projects, repositories, code_files (fields.content.rag for RAG)
repositories:
base_dir: "data/repositories"
repos: []
state:
dir: state
format: yaml
cleanup_interval: 60
controlplane:
base_url: "${FLUID_CONTROLPLANE_HTTP_BASE}"
api_version: "v1"
parameters:
organization_uuid: "${FLUID_CONTROLPLANE_ORGANIZATION_UUID}"
token: "${FLUID_CONTROLPLANE_TOKEN}"
```

Operational tuning (entities, intervals, repo list) can also come from control plane **`runtime_config`**.

## Commands

```bash
./scripts/install-git-hooks.sh
git submodule update --init --recursive
cp config/probe.example.yml config/probe.yml
cp env.secrets.example env.secrets
source env.secrets
make dev # go run ./cmd
make test
make fmt
```

Release: semver tag without `v`, matching `cmd/version.go` → `ghcr.io/fluid-pub/probe-gitlab:<tag>`.

## Standards

**Naming:** packages lowercase (`probe`, `config`, `gitlab`); exported types PascalCase; YAML field names snake_case per schema.

**Errors:** wrap with `fmt.Errorf("...: %w", err)`; never ignore errors without reason.

**Logging:** English messages via `log` / `log.Printf`.

**Git operations:** only **go-git** in `internal/repositories/sync.go`; auth via `plumbing/transport/http.BasicAuth` (`oauth2` + PAT for GitLab HTTPS).

**Secrets:** never commit tokens; use `env.secrets` and `${VAR}` in YAML.

## Boundaries

- **Always:** extend behavior via new or updated **entities** and probe-core hooks; keep generic examples in `probe.example.yml`; update `CHANGELOG.md` for notable changes.
- **Ask first:** new `go.mod` dependencies; schema-breaking field renames; changing control plane ingest shape.
- **Never:** `exec.Command("git", ...)`; customer-specific paths in versioned files; WebSocket control plane transport; commit `state/`, `data/`, or `probe.yml`.

## References

- Human-oriented: [README.md](README.md)
- Workspace rule (monorepo): `probes-no-git-cli-docker.mdc` — no git CLI in probe images
- Sibling reference: `fluid-pub/probe-confluence` (same probe-core + HTTP pattern)
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Changelog — probe-gitlab

All notable changes to **fluid-pub/probe-gitlab** are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

Tag naming: `0.y.z` (no `v` prefix). Align `cmd/version.go` with the tag before release.

## [Unreleased]

### Changed

- Repository sync uses **go-git** instead of the `git` CLI (no `git` binary required in the distroless image).

## [0.1.0] - 2026-05-23

### Added

- **probe-core** integration: entity-based lifecycle (`users`, `groups`, `projects`, `repositories`, `code_files`), HTTP control plane (`/probes`), `runtime_config` merge, schema in image.
- CI/CD via `fluid-pub/actions` (test, release on semver tag, GHCR image `ghcr.io/fluid-pub/probe-gitlab`).
- Git repository sync and `code_files` indexing for RAG.

### Changed

- Configuration uses `state` (replacing `output`) and per-entity `refresh_interval` (replacing global probe refresh).
Loading
Loading