recsys-pipeline-architect

Name: recsys-pipeline-architect
Author: affaan-m

Design composable Top-K recommendation, ranking, and feed pipelines for user-context decisions.

Stars

★ 286,082

Source

GitHub

Updated

2026-10-22

// securityLow risk

Prompt only, no code
Open source
Community-trusted· 286.1k

Running security audit…

Credentials
Network egress
Code execution
Data access
Supply chain

// install

Copy the install command and let the AI configure it · recommended for beginners

Please install the "recsys-pipeline-architect" skill from askskill:
1. Download https://raw.githubusercontent.com/affaan-m/ECC/main/skills/recsys-pipeline-architect/SKILL.md
2. Save it as ~/.claude/skills/recsys-pipeline-architect/SKILL.md
3. Reload skills and tell me it's ready

// download

Download SKILL.mdInstall manifest ↗

// examples

Design a social feed ranking pipeline

Prompt

Using the Source→Hydrator→Filter→Scorer→Selector→SideEffect six-stage framework, design a recommendation pipeline for a social product's For You feed. The goals are to improve session time and engagement while controlling low-quality content, author overexposure, and freshness decay. Output the responsibility, inputs/outputs, candidate features, scoring signals, Top-K selection strategy, and pre-launch metrics for each stage.

Expected

A structured feed architecture plan with six-stage design, ranking logic, and monitoring metrics.

Design a RAG reranking pipeline

Prompt

I am building an enterprise knowledge-base QA system. Based on the six-stage framework, design a RAG retrieval reranking pipeline that selects the most relevant Top-K document chunks for a user query plus retrieved context. Explain retrieval sources, feature hydration, filtering rules, relevance scoring, deduplication and diversity selection, plus logging and feedback loop design.

Expected

A RAG reranking pipeline design and optimization plan for knowledge-base QA.

Design a notification prioritization system

Prompt

Design a notification prioritization system for a mobile app using the Source→Hydrator→Filter→Scorer→Selector→SideEffect framework to choose the most worthwhile Top-K notifications for a given user and context. Balance click-through rate, conversion, frequency caps, user fatigue, and business priority, and provide an actionable split between rules and models.

Expected

A notification ranking system plan detailing stage strategies, constraints, and delivery feedback loops.

// docs

recsys-pipeline-architect

A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. It encodes the six-stage pattern — Source → Hydrator → Filter → Scorer → Selector → SideEffect — popularized by xAI's open-sourced For You algorithm (Apache 2.0). This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original.

Upstream: https://github.com/mturac/recsys-pipeline-architect

When to Use

User wants to build any system that picks "the top K items for a user/context"
User asks "how should I rank X" or describes a feed/personalization problem
User has a scoring function and needs the pipeline plumbing around it
User wants to migrate from a single relevance score to multi-action prediction with tunable weights
User is wrapping an LLM/ML scorer and needs filters, hydrators, side-effects, and a runnable scaffold in their stack (TypeScript / Go / Python)
Triggers: "recommendation system", "feed algorithm", "ranking pipeline", "for you feed", "candidate pipeline", "content recommender", "pipeline architecture for recsys", "RAG retrieval reranker"

When NOT to Use

Model architecture work (transformer design, two-tower retrieval, embedding training) — this skill is plumbing around the model, not the model itself
Pure ML training pipelines — the scoring function is the user's responsibility
Operating a deployed pipeline (monitoring, autoscaling) — out of scope

The six-stage framework

#	Stage	Job	Parallel?
1	Source	Fetch candidates from one or more origins	Yes — multiple sources run in parallel
2	Hydrator	Enrich each candidate with metadata needed for filtering and scoring	Yes — independent hydrators run in parallel
3	Filter	Drop candidates that should never be shown (blocked, expired, duplicate, ineligible)	Sequential — each filter sees fewer items
4	Scorer	Assign each surviving candidate one or more scores	Sequential — later scorers see earlier scores
5	Selector	Sort by final score, return top K	Single op
6	SideEffect	Cache served IDs, log impressions, emit events, update counters	Async — must never block the response

Why this exact order

Sources before hydration: know what candidates exist before paying to enrich them
Hydration before filtering: many filters need metadata the source did not provide
Filtering before scoring: scoring is the expensive stage; drop the ineligible first
Scorer chain (not single scorer): real systems compose ML scoring + diversity reranking + business rules
Selector after scoring: keeps scoring deterministic and cacheable
SideEffects last and async: side effects must never block the user response

Workflow when invoked

Walk the user through these eight steps:

Clarify the use case (one round, three questions): items being ranked? input context? language/runtime?
Identify the candidate sources: usually in-network (followed/owned/subscribed) + out-of-network (ML retrieval / trending / similar-to-liked)
List required hydrations: for each filter and scorer, what data does it need that the source did not provide?
List the filters: duplicate, self, age, block/mute, previously-served, eligibility. Order matters — cheap before expensive.
Design the scorer chain: primary (ML) → combiner (multi-action with weights) → diversity → business rules
Selector: sort descending by final score, take top K (or stratified mix for in-network/out-of-network)
SideEffects: cache served IDs, emit impression events, update counters, log analytics — all fire-and-forget
Generate the scaffold in the user's stack