Rewrite of the Worker in Go by ryzmae · Pull Request #1 · bueapp/worker

ryzmae · 2025-11-23T15:07:26Z

This allows for a faster compilation time also this is much better to maintain rather than rust which was overkill.

- Deleted Cargo.toml, Dockerfile, and all source files including config, dlq, flush, models, ttl_index, tuning, and main logic. - This commit removes the entire codebase, effectively archiving the project.

- Added configuration loading from environment variables with default values. - Created database connection pool and ensured necessary tables exist. - Implemented bulk insert functions for event logs, guardian logs, and join logs. - Developed a dead letter queue (DLQ) reprocessor for handling failed logs. - Introduced flushing mechanism to transfer logs from Redis to PostgreSQL. - Added models for event logs, guardian logs, and join logs with JSON serialization. - Implemented Redis client for interacting with Redis data structures. - Created tuning logic to dynamically adjust batch size and interval based on queue length. - Comprehensive unit tests for all new functionalities to ensure reliability.

Copilot

Pull request overview

This PR represents a complete rewrite of the worker service from Rust to Go, transitioning from MongoDB to PostgreSQL as the primary database. The change aims to improve compilation times and maintainability.

Key Changes:

Language Migration: Full rewrite from Rust to Go
Database Change: MongoDB replaced with PostgreSQL
Architecture: Maintains similar worker pattern with Redis queues, batch processing, and DLQ support
Test Coverage: Comprehensive unit tests added for all major components

Reviewed changes

Copilot reviewed 26 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
go.mod	Go module configuration with invalid version 1.25.3
cmd/worker/main.go	New main entry point with graceful shutdown and concurrent log flushing
internal/config/config.go	Configuration management with environment variables
internal/database/database.go	PostgreSQL implementation replacing MongoDB
internal/redis/redis.go	Redis client wrapper for list operations
internal/flush/flush.go	Core flush logic for batch processing
internal/dlq/dlq.go	Dead letter queue reprocessor
internal/tuning/tuning.go	Adaptive batch size and interval tuning
internal/models/models.go	Data models for EventLog, GuardianLog, JoinLog
Dockerfile	Multi-stage Go build replacing Rust build
README.md	Updated documentation for Go implementation
Makefile	New build, test, and development targets

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-23T15:09:46Z

internal/database/database.go

+
+	batch := &pgx.Batch{}
+	for _, log := range logs {
+		dataJSON, _ := json.Marshal(log["data"])


JSON marshaling errors in bulk insert functions are silently ignored with _. This could lead to data loss. Consider logging these errors or handling them appropriately.

Copilot · 2025-11-23T15:09:47Z

internal/database/database.go

+	if config.ConnConfig.TLSConfig == nil {
+		config.ConnConfig.TLSConfig = nil // Disable TLS
+	}


The code setting TLS config to nil when it's already nil is redundant. Consider removing this or adding a proper comment explaining the purpose of this check.

Suggested change

if config.ConnConfig.TLSConfig == nil {

config.ConnConfig.TLSConfig = nil // Disable TLS

}

// If TLSConfig is nil, TLS is disabled (handled by pgx)

Copilot · 2025-11-23T15:09:47Z

README.md

+
+## Prerequisites
+
+- Go 1.23+


The README states "Go 1.23+" as a prerequisite, but go.mod specifies go 1.25.3 which doesn't exist. These should be consistent and use a valid Go version.

Suggested change

- Go 1.23+

- Go 1.22+

Copilot · 2025-11-23T15:09:47Z

Dockerfile

@@ -1,30 +1,25 @@
-# Builder stage
-FROM rust:1.85-slim as builder
+FROM golang:1.23-alpine AS builder


The Dockerfile uses golang:1.23-alpine which is correct, but this is inconsistent with go.mod which specifies go 1.25.3. Use a valid and consistent Go version across all files.

Suggested change

FROM golang:1.23-alpine AS builder

FROM golang:1.25.3-alpine AS builder

Copilot · 2025-11-23T15:09:48Z

internal/flush/flush.go

+func FlushLogs(
+	ctx context.Context,
+	redisClient RedisClient,
+	db interface{},


The db parameter is declared but never used in the FlushLogs function. Either remove it or document why it's kept for interface compatibility.

Suggested change

db interface{},

Copilot · 2025-11-23T15:09:48Z

internal/flush/flush.go

+	if err != nil {
+		// On failure, push to DLQ
+		for _, logData := range parsedLogs {
+			jsonBytes, _ := json.Marshal(logData)


Error handling silently ignores JSON marshaling errors with _. This could hide data corruption issues. Consider logging the error or returning it.

Copilot · 2025-11-23T15:09:48Z

internal/dlq/dlq.go

+		log.Error().Err(err).Msg("Failed to reprocess DLQ logs, pushing back")
+		// Push back to DLQ
+		for _, logData := range parsedLogs {
+			jsonBytes, _ := json.Marshal(logData)


Similarly, JSON marshaling errors are silently ignored with _. Consider logging these errors to track data issues.

ryzmae · 2025-11-30T17:56:47Z

@copilot import {
pgTable,
text,
timestamp,
jsonb,
uuid,
index,
boolean,
integer,
} from "drizzle-orm/pg-core";

export const events = pgTable(
"events",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
action: text("action").notNull(),
actionThreadId: text("action_thread_id"),
loggedAt: timestamp("logged_at").notNull().defaultNow(),

// Flattened values (instead of nested object)
actorId: text("actor_id"),
channelId: text("channel_id"),
targetId: text("target_id"),
messageId: text("message_id"),
reason: text("reason"),
duration: integer("duration"),
oldValue: jsonb("old_value"),
newValue: jsonb("new_value"),
url: text("url"),

createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),

},
(table) => [
index("events_guild_id_idx").on(table.guildId),
index("events_action_idx").on(table.action),
index("events_logged_at_idx").on(table.loggedAt),
index("events_actor_id_idx").on(table.actorId),
index("events_target_id_idx").on(table.targetId),
index("events_action_thread_id_idx").on(table.actionThreadId),
// Composite indexes for common query patterns
index("events_guild_logged_idx").on(table.guildId, table.loggedAt),
index("events_guild_action_logged_idx").on(
table.guildId,
table.action,
table.loggedAt,
),
index("events_guild_actor_logged_idx").on(
table.guildId,
table.actorId,
table.loggedAt,
),
index("events_guild_target_logged_idx").on(
table.guildId,
table.targetId,
table.loggedAt,
),
],
);

export const actionThreads = pgTable(
"action_threads",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
actionThreadId: text("action_thread_id").notNull().unique(),
discordThreadId: text("discord_thread_id").notNull(),
targetUserId: text("target_user_id").notNull(),
reason: text("reason").notNull(),
status: text("status").notNull().default("OPEN"), // OPEN, CLOSED, ARCHIVED, INVESTIGATING, RESOLVED
resolvedAt: timestamp("resolved_at"),
resolvedByUserId: text("resolved_by_user_id"),
createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),
},
(table) => [
index("action_threads_guild_id_idx").on(table.guildId),
index("action_threads_discord_thread_id_idx").on(table.discordThreadId),
index("action_threads_target_user_id_idx").on(table.targetUserId),
index("action_threads_action_thread_id_idx").on(table.actionThreadId),
],
);

export const actionThreadsSettings = pgTable(
"action_threads_settings",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull().unique(),
channelId: text("channel_id").notNull(),
eventConfig: jsonb("event_config").notNull(), // { USER_BAN: true, USER_KICK: false, ... }
createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),
},
(table) => [index("action_threads_settings_guild_id_idx").on(table.guildId)],
);

export const loggingSettings = pgTable(
"logging_settings",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull().unique(),
enabled: boolean("enabled").notNull().default(false),
channelId: text("channel_id").notNull(),
eventConfig: jsonb("event_config").notNull(), // { MemberKick: { enabled: true }, ... }
createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),
},
(table) => [index("logging_settings_guild_id_idx").on(table.guildId)],
);

export const guardianLogs = pgTable(
"guardian_logs",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
action: text("action").notNull(),
performedBy: text("performed_by").notNull(),
target: text("target"),
reason: text("reason"),
metadata: jsonb("metadata"),
createdAt: timestamp("created_at").notNull().defaultNow(),
},
(table) => [
index("guardian_logs_guild_id_idx").on(table.guildId),
index("guardian_logs_action_idx").on(table.action),
index("guardian_logs_performed_by_idx").on(table.performedBy),
],
);

export const joinLogs = pgTable(
"join_logs",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
userId: text("user_id").notNull(),
action: text("action").notNull(), // 'join' | 'leave'
createdAt: timestamp("created_at").notNull().defaultNow(),
},
(table) => [
index("join_logs_guild_id_idx").on(table.guildId),
index("join_logs_user_id_idx").on(table.userId),
index("join_logs_action_idx").on(table.action),
],
);

export const aggregatedDailyStats = pgTable(
"aggregated_daily_stats",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
date: timestamp("date").notNull(),
stats: jsonb("stats").notNull(), // Aggregate data
createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),
},
(table) => [
index("aggregated_daily_stats_guild_id_idx").on(table.guildId),
index("aggregated_daily_stats_date_idx").on(table.date),
// Composite index for common query pattern
index("aggregated_daily_stats_guild_date_idx").on(
table.guildId,
table.date,
),
],
);

export const warnings = pgTable(
"warnings",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),

// Warning details
warnId: text("warn_id").notNull().unique(), // Human-readable ID like "W-1234"
userId: text("user_id").notNull(), // User who received the warning
moderatorId: text("moderator_id").notNull(), // Moderator who issued it
reason: text("reason").notNull(),

// Status tracking
active: boolean("active").notNull().default(true), // Can be pardoned/removed
pardonedAt: timestamp("pardoned_at"),
pardonedBy: text("pardoned_by"),
pardonReason: text("pardon_reason"),

// Auto-punishment tracking
punishmentTriggered: text("punishment_triggered"), // What punishment was triggered (if any)

createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),

},
(table) => [
index("warnings_guild_id_idx").on(table.guildId),
index("warnings_user_id_idx").on(table.userId),
index("warnings_moderator_id_idx").on(table.moderatorId),
index("warnings_warn_id_idx").on(table.warnId),
index("warnings_active_idx").on(table.active),
index("warnings_guild_user_idx").on(table.guildId, table.userId),
],
);

/**

Punishment actions that can be triggered
*/
export type PunishmentAction =
| "TIMEOUT_5M"
| "TIMEOUT_10M"
| "TIMEOUT_1H"
| "TIMEOUT_1D"
| "TIMEOUT_1W"
| "KICK"
| "BAN";

/**

Guild configuration for automatic punishments at warn thresholds
*/
export const warnPunishments = pgTable(
"warn_punishments",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
warnCount: integer("warn_count").notNull(), // Number of warns to trigger this
action: text("action").notNull(), // PunishmentAction type
duration: integer("duration"),
deleteMessageDays: integer("delete_message_days").default(0),

createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),
},
(table) => [
index("warn_punishments_guild_id_idx").on(table.guildId),
index("warn_punishments_warn_count_idx").on(table.warnCount),
],
);

/**

Private moderator notes on users
Supports markdown formatting for rich text
*/
export const moderatorNotes = pgTable(
"moderator_notes",
{
id: uuid("id").primaryKey().defaultRandom(),
guildId: text("guild_id").notNull(),
noteId: text("note_id").notNull().unique(), // Human-readable ID like "N-ABC123"
userId: text("user_id").notNull(), // User the note is about
authorId: text("author_id").notNull(), // Moderator who created the note
content: text("content").notNull(), // Markdown-supported content
editedAt: timestamp("edited_at"),
editedBy: text("edited_by"),

createdAt: timestamp("created_at").notNull().defaultNow(),
updatedAt: timestamp("updated_at").notNull().defaultNow(),
},
(table) => [
index("moderator_notes_guild_id_idx").on(table.guildId),
index("moderator_notes_user_id_idx").on(table.userId),
index("moderator_notes_author_id_idx").on(table.authorId),
index("moderator_notes_note_id_idx").on(table.noteId),
index("moderator_notes_guild_user_idx").on(table.guildId, table.userId),
],
);

export type ModeratorNote = typeof moderatorNotes.$inferSelect;
export type Event = typeof events.$inferSelect;
export type ActionThread = typeof actionThreads.$inferSelect;
export type ActionThreadsSetting = typeof actionThreadsSettings.$inferSelect;
export type LoggingSetting = typeof loggingSettings.$inferSelect;
export type GuardianLog = typeof guardianLogs.$inferSelect;
export type JoinLog = typeof joinLogs.$inferSelect;
export type AggregatedDailyStat = typeof aggregatedDailyStats.$inferSelect;
export type Warning = typeof warnings.$inferSelect;
export type WarnPunishment = typeof warnPunishments.$inferSelect;

this is my current version of the drizzle schema as we discussed earily i only want to fetch events from redis and then push into the database in bulk

Copilot · 2025-11-30T17:56:55Z

@ryzmae I've opened a new pull request, #2, to work on those changes. Once the pull request is ready, I'll request review from you.

…rocessing

ryzmae added 3 commits November 23, 2025 15:05

fix: Add .DS_Store to .gitignore

df17a2c

Remove project files and codebase for bue-worker

0bd3ed9

- Deleted Cargo.toml, Dockerfile, and all source files including config, dlq, flush, models, ttl_index, tuning, and main logic. - This commit removes the entire codebase, effectively archiving the project.

ryzmae requested a review from Copilot November 23, 2025 15:07

Copilot started reviewing on behalf of ryzmae November 23, 2025 15:07 View session

Copilot finished reviewing on behalf of ryzmae November 23, 2025 15:09

Copilot AI reviewed Nov 23, 2025

View reviewed changes

Copilot AI mentioned this pull request Nov 30, 2025

Align worker with Drizzle events schema, simplify to events-only processing #2

Draft

feat: Refactor logging and database interaction for improved stream p…

ad2becb

…rocessing

ryzmae merged commit c6ef94b into master Nov 30, 2025

ryzmae deleted the go-rewrite branch November 30, 2025 18:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite of the Worker in Go#1

Rewrite of the Worker in Go#1
ryzmae merged 4 commits intomasterfrom
go-rewrite

ryzmae commented Nov 23, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

Copilot AI Nov 23, 2025

Uh oh!

ryzmae commented Nov 30, 2025

Uh oh!

Copilot AI commented Nov 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	FROM golang:1.23-alpine AS builder
	FROM golang:1.25.3-alpine AS builder

Conversation

ryzmae commented Nov 23, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes:

Reviewed changes

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

ryzmae commented Nov 30, 2025

Uh oh!

Copilot AI commented Nov 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants