LycheeMem

🗺️ Semantic Memory — Compact Semantic Memory

Semantic memory is organised around typed, action-annotated MemoryRecords. The storage layer is SQLite (FTS5 full-text search) + LanceDB (vector index).

Memory Record Types

Each memory entry is stored as a MemoryRecord. The memory_type field distinguishes seven semantic categories:

| Type | Description | |------|-------------| | fact | Objective facts about the user, environment, or world | | preference | User preferences (style, habits, likes/dislikes) | | event | Specific events that have occurred | | constraint | Conditions that must be respected | | procedure | Reusable step-by-step procedures / methods | | failure_pattern | Previously failed action paths and their causes | | tool_affordance | Capabilities and applicable scenarios of tools/APIs |

Beyond text, every MemoryRecord carries action-facing metadata (tool_tags, constraint_tags, failure_tags, affordance_tags) and usage statistics (retrieval_count, action_success_count, etc.) to seed future reinforcement-learning signals.

Related MemoryRecords can be fused online by the Record Fusion Engine into a denser CompositeRecord; composite entries are ranked above source fragments during retrieval.

Four-Module Pipeline

Module 1: Compact Semantic Encoding

A single-pass pipeline that converts conversation turns into a list of MemoryRecords:

1. Typed extraction — LLM extracts self-contained facts and assigns a semantic category to each record.
2. Decontextualization — Pronouns and context-dependent phrases are expanded into full expressions, so each record is understandable without the original dialogue.
3. Action metadata annotation — LLM annotates each record with memory_type, tool_tags, constraint_tags, failure_tags, affordance_tags, and other structured labels.

record_id = SHA256(normalized_text) — naturally idempotent; duplicate content is deduplicated automatically.

Module 2: Record Fusion

Triggered online after each consolidation:

1. FTS detects existing entries whose text is similar to the new records (candidate pool).
2. LLM judges whether the candidate pool is worth merging (synthesis_judge).
3. If yes, LLM executes the merge and produces a CompositeRecord written to both SQLite and LanceDB; original records are retained.

Module 3: Action-Aware Search Planning

Before retrieval, ActionAwareRetrievalPlanner analyses the user query and emits a SearchPlan:

• mode: answer (factual Q&A) / action (needs execution) / mixed
• semantic_queries: content-facing search terms
• pragmatic_queries: action/tool/constraint-facing search terms
• tool_hints: tools likely needed for this request
• required_constraints: constraints that are missing
• missing_slots: parameters / slots that are absent

The plan drives five-channel recall:

1. FTS channel — SQLite FTS5 keyword recall over MemoryRecord + CompositeRecord
2. Semantic vector channel — LanceDB ANN over semantic_text embeddings
3. Normalised vector channel — LanceDB ANN over normalized_text embeddings (for pragmatic queries)
4. Tag filter channel — exact filter by tool_hints / constraint_tags
5. Temporal channel — filter by SearchPlan.temporal_filter time window

Module 4: Multi-Dimensional Scorer

Candidates from all channels are de-duplicated and ranked by MemoryScorer using a weighted linear combination:

$$\text{Score} = \alpha \cdot S_\text{sem} + \beta \cdot S_\text{action} + \gamma \cdot S_\text{temporal} + \delta \cdot S_\text{recency} + \eta \cdot S_\text{evidence} - \lambda \cdot C_\text{token}$$

| Weight | Meaning | Default | |--------|---------|---------| | α | SemanticRelevance (vector distance -> similarity) | 0.30 | | β | ActionUtility (tag match score, mode-aware) | 0.25 | | γ | TemporalFit (temporal reference match) | 0.15 | | δ | Recency (memory freshness) | 0.10 | | η | EvidenceDensity (evidence span density) | 0.10 | | λ | TokenCost penalty (text length penalty) | 0.10 |

🛠️ Procedural Memory — Skill Store

The skill store preserves reusable how-to knowledge as structured skill entries, each carrying:

• Intent — a short description of what the skill does.
• doc_markdown — a full Markdown document describing the procedure, commands, parameters, and caveats.
• Embedding — a dense vector of the intent text, used for similarity search.
• Metadata — usage counters, last-used timestamp, preconditions.

Skill retrieval uses HyDE (Hypothetical Document Embeddings): the query is first expanded into a hypothetical ideal answer by the LLM, then that draft text is embedded to produce a query vector that matches well against stored procedure descriptions, even when the user's original phrasing is vague.

⚙️ Pipeline

Every request passes through a fixed sequence of five agents. Four are synchronous stages in the LangGraph pipeline; one is a background post-processing task.

Stage 1 — WMManager

Rule-based agent (no LLM prompt). Appends the user turn to the session log, counts tokens, and fires compression if either threshold is crossed. Produces compressed_history and raw_recent_turns for downstream stages.

Stage 2 — SearchCoordinator

ActionAwareRetrievalPlanner first analyses the user query and produces a SearchPlan containing mode, semantic_queries, pragmatic_queries, tool_hints, and more. Five parallel recall channels (FTS full-text, semantic vector, normalised vector, tag filter, temporal filter) then query SQLite + LanceDB, and the resulting candidates are ranked by the six-dimensional Scorer formula before being merged into background_context. Skill sub-queries use HyDE embe