deployment-manager/README.md

Deployment Manager

A Kubernetes deployment manager that reads repository URLs from a database and automatically deploys them to Kubernetes clusters.

Features

• Automated Deployment: Clone Git repositories, build Docker images, and deploy to Kubernetes
• Multi-Language Support: Supports Node.js and Python applications
• Real-time Events: Server-Sent Events (SSE) for real-time deployment status updates
• REST API: Full REST API for managing repositories and deployments
• Worker Pool: Concurrent deployment processing with configurable worker pool size
• Reconciliation Loop: Automatic retry and status reconciliation
• Health Checks: Built-in health check endpoints

Architecture

The deployment manager consists of several components:

• API Server: HTTP server with REST endpoints and SSE support
• Worker Pool: Concurrent workers that process deployment jobs
• Reconciler: Background process that ensures desired state matches actual state
• Event Bus: Pub/sub system for real-time event streaming
• Database: SQLite database for repository metadata
• Executor: Command execution and Docker operations
• Kubernetes Client: Kubernetes resource management

Quick Start

Prerequisites

• Go 1.21+
• Docker
• kubectl configured with cluster access
• Git

Building

# Build the binary
go build ./cmd/manager

# Or build with Docker
docker build -t deployment-manager .

Running

# Run locally
./manager

# Run with Docker
docker run -p 8080:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ~/.kube:/root/.kube \
  deployment-manager

Environment Variables

• DB_PATH: Path to SQLite database (default: ./manager.db)
• MAX_WORKERS: Number of concurrent workers (default: 2)
• RECONCILE_TICK: Reconciliation interval in seconds (default: 2)
• HTTP_PORT: HTTP server port (default: 8080)

API Endpoints

Repositories

• GET /api/repos - List all repositories
• GET /api/repos?user_id=xxx - List repositories by user
• POST /api/repos - Create a new repository
• GET /api/repos/{id} - Get repository details
• DELETE /api/repos/{id} - Delete a repository
• POST /api/repos/{id}/stop - Stop deployment
• POST /api/repos/{id}/restart - Restart deployment

Events

• GET /events - Server-Sent Events stream for real-time updates

Health

• GET /health - Health check endpoint

API Usage Examples

Create a Repository

curl -X POST http://localhost:8080/api/repos \
  -H "Content-Type: application/json" \
  -d '{
    "repo_url": "https://github.com/user/my-node-app.git",
    "user_id": "user123",
    "type": "nodejs"
  }'

List Repositories

curl http://localhost:8080/api/repos

Stream Events

curl -N http://localhost:8080/events

Repository Types

Node.js Applications

Expected structure:

• package.json with dependencies
• Entry point defined in package.json (default: index.js)
• Exposes port 3000

Python Applications

Expected structure:

• requirements.txt with dependencies
• app.py as entry point
• Exposes port 8000

Deployment Process

1. Repository Creation: Add repository via API
2. Job Queuing: Reconciler detects need_to_deploy status
3. Cloning: Worker clones the Git repository
4. Dockerfile Generation: Auto-generates appropriate Dockerfile
5. Image Build: Builds and pushes Docker image
6. Kubernetes Deployment: Applies Kubernetes manifests
7. Status Update: Updates repository status to deployed

Kubernetes Resources

The deployment manager creates the following Kubernetes resources:

• Deployment: Manages pod replicas and updates
• Service: Exposes the application internally
• ConfigMap: Stores repository metadata
• Ingress: External access (if ingress controller is available)

Event Types

• repo_created: New repository added
• repo_updated: Repository status changed
• repo_deleted: Repository deleted
• deploy_started: Deployment process started
• deploy_success: Deployment completed successfully
• deploy_error: Deployment failed
• log: Real-time log messages

Configuration

Database Schema

The SQLite database contains a repos table with the following schema:

CREATE TABLE repos (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  repo_url TEXT NOT NULL,
  status TEXT NOT NULL,
  user_id TEXT NOT NULL,
  type TEXT NOT NULL, -- nodejs | python
  image_tag TEXT,
  last_error TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
  updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

Status Transitions

From	To	Action
need_to_deploy	deploying	enqueue job
deploying	deployed	success
deploying	err	failure
deployed	stopped	scale 0
stopped	restarting	scale up
any	deleted	delete k8s resources

Development

Project Structure

deployment-manager/
├── cmd/manager/          # Application entry point
├── internal/
│   ├── api/             # HTTP server and handlers
│   ├── db/              # Database operations
│   ├── events/          # Event bus and types
│   ├── executor/        # Command execution and Docker
│   ├── k8s/             # Kubernetes client
│   ├── model/           # Data models
│   ├── reconciler/      # Reconciliation logic
│   └── worker/          # Deployment workers
├── migrations/          # Database migrations
├── manifests/           # K8s manifest templates
└── Dockerfile

Running Tests

go test ./...

Building for Production

# Build optimized binary
CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -o manager ./cmd/manager

# Build Docker image
docker build -t deployment-manager:latest .

Troubleshooting

Common Issues

1. Docker socket not accessible: Mount /var/run/docker.sock when running in Docker
2. kubectl not configured: Ensure ~/.kube/config is mounted and accessible
3. Database permissions: Ensure the application can write to the database file
4. Port conflicts: Change HTTP_PORT if 8080 is already in use

Logs

Enable debug logging by setting the log level:

LOG_LEVEL=debug ./manager

Health Checks

Monitor the deployment manager health:

curl http://localhost:8080/health

License

This project is licensed under the MIT License.