Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 138 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,159 @@
# indy-metadata-service
# Indy Metadata Service

A Quarkus-based microservice for managing metadata operations in the Indy artifact repository system. This service handles metadata file management, special path operations, and provides REST APIs for metadata-related operations.

## Prerequisite
1. jdk11
2. mvn 3.6.2+
## Overview

## Configure services
The Indy Metadata Service is part of the CommonJava Indy ecosystem and provides:

To make it run, Infinispan and Kafka need to be configured. Please see the example in `application.yaml`.
- **Metadata Management**: Automatic cleanup and management of metadata files (e.g., `maven-metadata.xml`, `package.json`)
- **Group-Aware Operations**: Metadata updates propagate through repository group hierarchies
- **Event-Driven Architecture**: Reactive processing of file events via Kafka
- **RESTful APIs**: HTTP endpoints for metadata operations
- **Multi-Package Support**: Support for Maven, NPM, and other package types

## Try it
## Architecture

### 1. Start the cache server and the message broker
### Core Components

It needs a remote cache server(Infinispan) as well as the message broker(Kafka) for event handler. To ease the setup, we have provided a `docker-compose.yml` file which start the containers and bind the network ports.
- **MetadataController**: Main business logic controller
- **MetadataHandler**: Handles metadata deletion and cleanup operations
- **FileEventConsumer**: Processes file events from Kafka for automatic metadata cleanup
- **MetadataResource**: JAX-RS REST endpoints

```
docker-compose up
```
## API Endpoints

### 2. Start gateway in debug mode
```
$ mvn quarkus:dev
### Metadata Operations

```http
GET /api/metadata/{packageType}/{type}/{name}/{path}
DELETE /api/metadata/{packageType}/{type}/{name}/{path}
GET /api/metadata/{packageType}/{type}/{name}
```

### 3. Verify the installation
**Parameters:**
- `packageType`: Package type (maven, npm, etc.)
- `type`: Store type (hosted, group, remote)
- `name`: Store name
- `path`: File path within the store

## Configuration

### Application Properties

Key configuration options in `application.yaml`:

```yaml
quarkus:
http:
port: 8081
read-timeout: 30m
limits:
max-body-size: 500M

kafka:
bootstrap:
servers: "localhost:9092"

storage-service-api/mp-rest/url: http://localhost
repo-service-api/mp-rest/url: http://localhost
```
http://localhost:8080/swagger-ui/

### Environment Variables

- `KAFKA_BOOTSTRAP_SERVERS`: Kafka broker addresses
- `STORAGE_SERVICE_API_MP_REST_URL`: Storage service URL
- `REPO_SERVICE_API_MP_REST_URL`: Repository service URL

## Development

### Prerequisites

- Java 11+
- Maven 3.6+
- Docker & Docker Compose (for local development)

### Building

```bash
# Compile and run tests
mvn clean compile test

# Package the application
mvn package

# Run in development mode
mvn quarkus:dev
```

### 4. Check the entries on cache server
### Docker Development

```bash
# Build the Docker image
docker build -f src/main/image/Dockerfile.jvm -t indy-metadata-service .

# Run with Docker Compose (includes Kafka)
docker-compose up
```
http://localhost:11222/console/

### Local Development Setup

1. Start Kafka and Zookeeper:
```bash
docker-compose up zookeeper kafka
```

2. Configure external services (storage and repository services)

3. Run the application:
```bash
mvn quarkus:dev
```

## Event Processing

The service processes two types of Kafka events and maintains consistency across repository group hierarchies:

> **Important**: When a metadata file is updated in any repository, the service automatically propagates these changes up through the entire group hierarchy. For example, if a POM file is uploaded to a hosted repository, the metadata will be regenerated not only for that repository but also for all groups that contain it, all the way up to the root group.

### File Events (`file-event` topic)
- Triggers metadata cleanup when POM files or package tarballs are uploaded/deleted
- Automatically clears corresponding metadata files in hosted repositories
- **Group Propagation**: Updates cascade through all repository groups containing the affected repository

### Promote Events (`promote-complete` topic)
- Handles promotion completion events
- Manages metadata updates across repository groups


## Monitoring & Observability

- **OpenTelemetry**: Distributed tracing support
- **Structured Logging**: JSON-formatted logs with configurable levels
- **Health Checks**: Built-in Quarkus health endpoints
- **Metrics**: Application metrics via Micrometer

## Testing

```bash
# Run unit tests
mvn test

# Run integration tests
mvn verify

# Run specific test class
mvn test -Dtest=MetadataResourceTest
```

## Dependencies

- **Quarkus**: Reactive framework and runtime
- **RESTEasy**: JAX-RS implementation
- **SmallRye Reactive Messaging**: Kafka integration
- **OpenTelemetry**: Observability
- **Maven Repository Metadata**: Maven-specific metadata handling

## License

Licensed under the Apache License, Version 2.0. See the [LICENSE](LICENSE) file for details.