S3-compatible file storage

Upload files, store on R2/MinIO/AWS, serve with signed URLs.

8 minmedium

Files don't belong in your database (rows balloon, backups get expensive). They belong in object storage — S3 or S3-compatible. Grit ships a storage battery that abstracts the provider, so MinIO locally and R2 or AWS in production feels the same to your code.

Why we need it

Three jobs object storage handles better than a DB:

Big binary data — images, PDFs, videos. Cheap per-GB, infinite scale, no DB bloat.
Direct serving — clients fetch files via CDN-backed URLs, not through your API.
Signed URLs — give time-limited download access without sharing credentials.

Where it lives

apps/api/internal/storage/
├── storage.go        ← Service: Upload, URL, Delete
├── handler.go        ← HTTP upload handler (multipart/form-data)
└── image.go          ← Resize, crop, thumbnail (uses imaging)

How it's implemented

apps/api/internal/storage/storage.go (simplified)

type Service struct {
  client *minio.Client    // works with AWS S3, R2, MinIO — they all speak S3
  bucket string
  baseURL string          // for serving (CDN domain in prod)
}

func New(endpoint, accessKey, secretKey, bucket string, useSSL bool) (*Service, error) {
  c, err := minio.New(endpoint, &minio.Options{
    Creds:  credentials.NewStaticV4(accessKey, secretKey, ""),
    Secure: useSSL,
  })
  if err != nil { return nil, err }
  return &Service{client: c, bucket: bucket}, nil
}

func (s *Service) Upload(ctx context.Context, key string, r io.Reader, size int64, mime string) error {
  _, err := s.client.PutObject(ctx, s.bucket, key, r, size, minio.PutObjectOptions{
    ContentType: mime,
  })
  return err
}

func (s *Service) URL(key string) string {
  return s.baseURL + "/" + key
}

func (s *Service) Delete(ctx context.Context, key string) error {
  return s.client.RemoveObject(ctx, s.bucket, key, minio.RemoveObjectOptions{})
}

func (s *Service) SignedURL(ctx context.Context, key string, ttl time.Duration) (string, error) {
  u, err := s.client.PresignedGetObject(ctx, s.bucket, key, ttl, nil)
  if err != nil { return "", err }
  return u.String(), nil
}

Same interface, three deploy targets:

Local dev: MinIO (via docker-compose).
Cheap prod: Cloudflare R2 (zero egress fees).
Big prod: AWS S3.

Same code; only the env vars change.

The upload handler

apps/api/internal/storage/handler.go

func (h *Handler) Upload(c *gin.Context) {
  fh, err := c.FormFile("file")
  if err != nil {
    c.JSON(400, gin.H{"error": gin.H{"code": "no_file"}})
    return
  }
  if fh.Size > 10*1024*1024 {  // 10 MB cap
    c.JSON(400, gin.H{"error": gin.H{"code": "too_large"}})
    return
  }
  src, err := fh.Open()
  if err != nil { c.JSON(500, gin.H{"error": gin.H{"code": "open_failed"}}); return }
  defer src.Close()

  key := uuid.NewString() + filepath.Ext(fh.Filename)
  mime := fh.Header.Get("Content-Type")

  if err := h.storage.Upload(c.Request.Context(), key, src, fh.Size, mime); err != nil {
    c.JSON(500, gin.H{"error": gin.H{"code": "upload_failed", "message": err.Error()}})
    return
  }

  // Record in DB so we can list / delete later
  upload := models.Upload{
    Key: key, OriginalName: fh.Filename, Size: fh.Size, Mime: mime,
    UserID: c.GetUint("user_id"),
  }
  h.db.Create(&upload)

  c.JSON(201, gin.H{"data": gin.H{
    "id":   upload.ID,
    "url":  h.storage.URL(key),
    "size": fh.Size,
  }})
}

Three details worth absorbing:

Random key (uuid + ext) — never trust the client filename. ../../../etc/passwd is a real attack.
Size cap upfront. Otherwise a malicious user can DOS your storage budget.
DB record. The S3 object is the bytes; theuploads table is the metadata. You need both to let users list / delete.

The frontend upload

async function uploadFile(file: File) {
  const form = new FormData()
  form.append('file', file)
  const res = await fetch('/api/uploads', {
    method: 'POST',
    body: form,                // do NOT set Content-Type — let the browser set it
    credentials: 'include',
  })
  return res.json()
}

Critical gotcha: don't set Content-Type manually. The browser must set it with the correct boundary string for multipart parsing. Override it and Gin can't read the file.

Image processing

apps/api/internal/storage/image.go (sketch)

func (s *Service) UploadImageWithThumbnail(ctx context.Context, src io.Reader, key string) error {
  img, _, err := image.Decode(src)
  if err != nil { return err }

  // Full size
  var full bytes.Buffer
  imaging.Encode(&full, img, imaging.JPEG, imaging.JPEGQuality(85))
  if err := s.Upload(ctx, key, &full, int64(full.Len()), "image/jpeg"); err != nil { return err }

  // 200px thumbnail
  thumb := imaging.Resize(img, 200, 0, imaging.Lanczos)
  var tBuf bytes.Buffer
  imaging.Encode(&tBuf, thumb, imaging.JPEG, imaging.JPEGQuality(75))
  return s.Upload(ctx, "thumbs/" + key, &tBuf, int64(tBuf.Len()), "image/jpeg")
}

Cheap thumbnails: resize once on upload, serve thumbs in lists, full size on detail pages. Saves the user's bandwidth.

Set CORS on your bucket if you serve directly. If the frontend fetches images from uploads.example.com, that domain needs the right Access-Control-Allow-Origin on its responses. MinIO has it open in dev; R2/S3 you configure once.

Signed URLs — when you need access control

Public files (avatars, blog images): just use storage.URL(key). No auth, served via CDN.

Private files (user's personal documents): the bucket should reject anonymous access. Use storage.SignedURL(key, 5*time.Minute) — that returns a long URL with a query-string signature. Valid for 5 minutes, then dies. Hand the URL to the frontend; the user's browser fetches the file directly without going through your API.

How to modify this battery

Increase the size cap — edit the fh.Size check in handler.go.
Restrict MIME types — add an allow-list: allowedMimes := map[string]bool{"image/jpeg": true, ...}.
Add virus scanning — pipe the upload through ClamAV before saving. Heavier; only do it if your threat model requires it.
Per-user quota — sum uploads.size by user_id, reject if over.

Local dev wiring

docker-compose.yml (excerpt)

minio:
  image: minio/minio:latest
  command: server /data --console-address ":9001"
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
  ports:
    - "9000:9000"   # API
    - "9001:9001"   # web console

STORAGE_ENDPOINT=localhost:9000
STORAGE_ACCESS_KEY=minioadmin
STORAGE_SECRET_KEY=minioadmin
STORAGE_BUCKET=uploads
STORAGE_USE_SSL=false

For prod, swap to STORAGE_ENDPOINT=<account-id>.r2.cloudflarestorage.com + Cloudflare R2 credentials. Zero code changes.

Quick check

A user uploads `report.pdf`. Where does the bytes go, and where does the metadata go?

Try it

Add file upload to the User profile (avatar):

Add an AvatarKey string field to the User model.
Add a route: POST /api/me/avatar that accepts a multipart file, uploads via the storage service, sets user.AvatarKey, deletes the old one if present.
Return the URL via storage.URL(user.AvatarKey).
Display it in the admin user list.
Try uploading a 20MB file — confirm you get 400 due to size cap.

What's next

Next lesson — Mail (Resend). Transactional email with editable templates — the most-needed external service after the DB.

Spot a typo? Have an idea?

Help us improve this lesson. One click opens a GitHub issue with the lesson URL pre-filled — suggest clearer wording, report a bug, or request more depth. The course keeps improving thanks to learners like you.

Suggest an improvement on GitHub

Previous lesson Next lesson