Engineering reference

How LearnixSai is built.

LearnixSai is a curation-first learning platform: type a topic and it assembles a structured course out of real videos, articles, docs and Q&A — not AI-generated lesson text. This is the full engineering reference: architecture, algorithms, the API, AI integration, and how it all deploys. Diagrams and flows are interactive — click around.

System topology

Four independently-hosted pieces, deliberately decoupled so no single failure takes everything down. Click a box to see what it is and why it's there.

BrowserLEARNERStatus + DocsVERCELGitHub ActionsCI/CDGeneration workerpg-boss · in-processNext.js AppHEROKU · DOCKERPostgreSQLHEROKU · app + pgbossOpenRouterHYBRID LLMsSearXNGORACLE FREE VMContent APIsYT · Wiki · SE · arXiv

Click any box to see what it is, where it runs, and why it's there. Animated lines show live request paths.

Design principle

Decouple by failure domain. The app, the search engine, and the status page each run on separate infrastructure. SearXNG going down degrades one source, not the app. The status page lives on Vercel so it can report an outage even while Heroku is on fire.

Design principle

One database, many roles. Postgres holds application data, the pg-boss job queue (its own schema), and the rate-limit table. No Redis, no extra managed services — fewer moving parts to pay for and monitor.

Architecture decisions & why

The choices that shaped the system — each with the context that forced it and the trade-off it carries. These are the load-bearing decisions.

01

Curate real content, don't generate lesson text

USP
ContextAI-generated course text is a commodity and often subtly wrong.
DecisionAssemble every course from real videos, articles, docs and Q&A. AI only classifies, structures, reranks and summarizes — it never writes the lesson.
Trade-offHarder engineering (search, routing, reranking) in exchange for trustworthy, source-backed learning.
02

Generation is a background job, not an HTTP request

pg-boss
ContextThe original pipeline ran for minutes inside an SSE request — slow, and it fell over under concurrent load.
DecisionPOST enqueues a durable pg-boss job and returns in ~50ms; the worker runs the pipeline and the frontend polls a job row.
Trade-offA polling endpoint + job table to maintain, but the request path is now instant and load-safe.
03

Run the worker in-process, not on a second dyno

cost
ContextA dedicated worker dyno doubles the Heroku bill.
Decisioninstrumentation.ts starts the pg-boss worker inside the web dyno at boot, capped by GENERATION_CONCURRENCY.
Trade-offWorker shares web-dyno resources — fine at this scale; the documented scale-up path is a dedicated dyno with the same code.
04

Hybrid models — free to classify, paid to build

latency
ContextFree OpenRouter models share a global rate limit → 429 storms and long latency under concurrency.
DecisionLatency-critical calls (structure, rerank) use a paid fast model; classify + summaries stay free.
Trade-off~$0.001–0.01 per course, in exchange for killing both the 429s and the wait.
05

Self-host SearXNG on a free VM instead of a paid search API

$0
ContextBrave Search needs a credit card; public SearXNG instances block datacenter IPs or disable JSON.
DecisionRun our own SearXNG on an Oracle Cloud always-free VM, referenced by SEARXNG_URL.
Trade-offWe own the uptime (hence this status page), but general web search costs $0 and needs no keys.
06

Keyless structured APIs over web scraping

reliability
ContextScraping is brittle and gets IP-blocked; some APIs (wikiHow, Reddit) failed outright from servers.
DecisionRoute to stable keyless APIs by topic class — Stack Exchange, dev.to, GitHub, HN, arXiv, Wikipedia.
Trade-offCoverage is bounded by which APIs exist, but results are reliable and require no auth.
07

Postgres-backed rate limiter, not in-memory

correctness
ContextAn in-memory map is per-dyno — useless the moment there's more than one process.
DecisionAtomic INSERT … ON CONFLICT upsert on a RateLimit table enforces limits globally.
Trade-offOne extra tiny write per limited action, for a limit that's actually correct across dynos.
08

Keep the status page on separate infrastructure

ops
ContextA status page that shares fate with the app can't tell you the app is down.
DecisionHost status + docs as their own repo on Vercel; it checks the app from the outside.
Trade-offA second repo/deploy to maintain, but honest, independent monitoring.

Course generation pipeline

Generation is a background job, not part of the HTTP request. The old design ran the whole multi-minute pipeline inside an SSE request — slow, and it broke under load. Step through the current flow:

Enqueue & return

~50 msHTTP 202

The request does the bare minimum: auth, a Postgres rate-limit check, create a Course (status generating) and a 1:1 GenerationJob, enqueue to pg-boss, and return { courseId }. No work happens in the request — it returns in milliseconds.

POST /api/courses/generate
  auth + DB rate-limit
  create Course(status:"generating") + GenerationJob
  boss.send("course-generation", { courseId, topic, level, userId })
  → 202 { courseId }
⚡

Why it's durable: pg-boss coordinates through Postgres with SKIP LOCKED. If the dyno restarts mid-job (Heroku cycles daily), the job is retried on next boot — nothing is lost, and it's safe even if the app scales to multiple web dynos.

Feature workflows

How each user-facing feature is actually built, end to end. Pick a feature:

The flagship loop. Covered in depth under Generation pipeline — in short: enqueue → classify → structure → parallel curation → persist, with the frontend polling a job row. The important design move is that nothing generative happens in the request; it's a durable background job with a global concurrency cap.

Algorithms

The non-obvious logic that makes curation actually good — routing, filtering, reranking, balancing, and racing.

lib/source-router.ts maps the topic class to a set of keyless APIs, so a cooking course doesn't get Stack Overflow and a Docker course doesn't get arXiv.

switch (topicClass) {
  developer_tech:    dev.to(tag) + HackerNews + StackOverflow + GitHub
  business_founder:  HackerNews + dev.to
  academic_science:  arXiv + Stack Exchange (physics/stats/…)
  history_humanities: Stack Exchange (history/philosophy/…)
  practical_skill:   Stack Exchange topic site (guitar→music.se,
                     cooking→cooking.se … keyword map)
}
// Wikipedia + SearXNG web search run for every class

Removed after live testing: wikiHow (search API returns 500) and Reddit (403 from datacenter IPs). Brave Search was rejected — it requires a credit card even on the free tier.

API system

The routes that matter, from generation to health. Click to expand request/response shapes.

AuthAuthenticated · Postgres rate-limited
Request{ topic, level, mode?, pathNodeId? }
Response202 { courseId } — returned in ~50ms; work happens in the background worker.
NotesCreates Course(status:generating) + GenerationJob, enqueues to pg-boss. No SSE.

AI integration

Every model call, why it's on the tier it's on, and the prompts behind them.

FREE tier

Cheap, simple, tolerant of latency

  • Topic classification — one label + a module estimate.
  • Article summaries — lazy, cached, off the critical path.
PAID tier

Fast, not globally rate-limited

  • Structure — course title + ordered modules.
  • Rerank — the per-module editor pass.

Why hybrid

Free OpenRouter models share a global account rate limit — running the whole pipeline on them meant 429 storms the moment two users generated at once, and long queues. Moving the latency-critical, on-the-hot-path calls (structure + rerank) to a paid fast model (Gemini Flash, ~$0.001–0.01/course) killed both the latency and the 429s, while classify + summaries stay free. Models are env-overridable via PIPELINE_PAID_MODELS, and the client cycles through a fallback list on error.

The prompts

classifier.ts

Topic → class + module estimate

curator.ts · structure

Class + topic → title + modules

curator.ts · buildRerankPrompt

Candidates → { keep: [indices] }

summary.ts

Article text → { summary, keyPoints }

quiz prompt

Module resources → questions

🔒

Keys never touch the client. All model calls are server-side; the OpenRouter key lives only in Heroku config vars. No tokens in cookies or the frontend bundle.

Data model

17 Prisma models on PostgreSQL, ID strategy cuid() throughout. Filter by domain and click any entity for its real fields, types and relations.

Relationship map

Everything hangs off User and Course. A course owns its modules and resources; a module optionally owns a quiz; progress and XP are per-user.

User ─┬─< Course ─┬─< Module ─┬─< Resource ─< ResourceProgress >─ User │ │ └── Quiz ─< QuizAttempt >─ User │ ├─── GenerationJob (1:1, poll target) │ └─< CourseRating >─ User ├─< XpTransaction (append-only XP ledger) ├─< UserAchievement (25 unlockables) ├─< LearningPath ─< PathNode ··> Course (skill trees) └─< Account / Session (NextAuth) RateLimit (standalone · cross-dyno limiter)

Entity explorer

User
14 fields
Course
15 fields
Module
6 fields
Resource
14 fields
GenerationJob
14 fields
ResourceProgress
9 fields
Quiz
4 fields
QuizAttempt
8 fields
LearningPath
7 fields
PathNode
9 fields
XpTransaction
6 fields
UserAchievement
5 fields
CourseRating
6 fields
Account
7 fields
Session
4 fields
VerificationToken
3 fields
RateLimit
3 fields

User

Core

The learner and their progression state. NextAuth credentials + gamification live on the same row.

idStringPK
emailString?UNIQUE
usernameString?UNIQUE
passwordString?
xpInt @default(0)
levelInt @default(1)
streakCountInt @default(0)
longestStreakInt @default(0)
lastActiveDateTime?
avatarString @default('default')
avatarFrameString @default('none')
titleString @default('Novice')
characterDataJson?
createdAtDateTime
Relations
coursesresourceProgressquizAttemptsxpTransactionsachievementscourseRatingslearningPathsaccountssessions

Status & enum values

These are string columns (no DB enums), so the valid values live in code. The ones that matter:

Course.status
generating · active · completed · failed — listings hide generating + failed.
GenerationJob.status
queued · running · done · failed
GenerationJob.step
classifying · structure · module · done
Resource.type
youtube · article · wikipedia · paper · ai_generated
ResourceProgress.status
not_started · in_progress · completed
topicClass
developer_tech · academic_science · business_founder · history_humanities · practical_skill
PathNode.status
locked · active · completed
Course.level
beginner · intermediate · advanced
🗄️

One database, three roles. These models live alongside the pg-boss queue (its own pgboss schema) and the RateLimit table on a single Heroku Postgres. Prisma 7 with the @prisma/adapter-pg driver adapter; migrations applied via prisma migrate deploy on release.

Deployment · CI/CD

The app ships through GitHub Actions on every push to main — no manual deploys. Step through the pipeline:

Push to main

GitHub

A merge/push to main triggers the deploy workflow in .github/workflows/deploy.yml. Feature branches don't deploy.

▲

This ops site deploys separately. It's its own repo on Vercel — pushing auto-deploys it. It was split out of the main repo because the parent's Tailwind/PostCSS config leaked into the subfolder build.

Deployment · servers

Where everything physically runs, and the constraints that shaped each host.

🟣

Next.js App + Worker

Heroku · Docker
Runtime
Standalone Docker image, App Router
Worker
pg-boss, in-process (no 2nd dyno)
Why not Vercel
Serverless can't host a long-lived worker
Deploy
GitHub Actions → migrate → release
🐘

PostgreSQL

Heroku Postgres
App data
All core + gamification models
Queue
pg-boss pgboss schema
Limiter
RateLimit table (cross-dyno)
Migrations
prisma migrate deploy on release
🔍

SearXNG

Oracle Cloud · always-free VM
Shape
VM.Standard.E2.1.Micro, Ubuntu
Cost
$0 / month (always-free tier)
Run
Docker, restart: unless-stopped
Caveat
Ephemeral public IP — reserve or update env if it changes
🟢

Status + Docs

Vercel
Role
Live status board + these docs
Isolation
Separate repo, off Heroku on purpose
Checks
/api/health + frontend + SearXNG
Alerts
UptimeRobot can watch /api/health
← Back to live statusLearnixSai · curation-first learning