akinus commented

Owner

Corten — Project Guide for OpenCode

What is Corten?

Corten is a Rust-based file tagging system with a FUSE virtual filesystem, semantic tag taxonomy, and built-in AI auto-tagging via Ollama. It is the spiritual successor to AkTags, rebuilt around the TMSU concept but extended with AI integration, a file watcher, and an MCP server.

The name comes from corten steel — a material that rusts intentionally, forming a protective oxide layer. Corten does the same for your filesystem: it imposes deliberate, protective structure on file chaos.

Corten is part of the AK metalworking ecosystem:

Iron — browser
Corten — file tagger (this project)
Corten Beams — cloud/server layer (future)
Anvil — global command toolkit (future)

Core Design Philosophy

The tag taxonomy is the memory. No opaque embeddings. No black box ML. Tag names + human-written descriptions are the only persistent knowledge store. AI decisions are always explainable because they reference visible, editable descriptions.
Every tag has a description. A tag is not just a word. It is a word + a human-readable statement of purpose. Example: tag reference, description Files I may want to look up later but don't need to act on now. This description is what the AI reads when deciding whether to apply a tag.
AI suggestions must include descriptions. When the AI suggests a new tag that doesn't exist yet, it must provide a proposed description. The user sees both the tag name and the AI's reasoning before accepting.
Corrections propagate forward automatically. When a user edits an AI suggestion or its description, that refined description influences all future tagging decisions without any explicit retraining. The taxonomy gets smarter over time naturally.
Any file type is supported. Corten extracts meaningful content from any file via a trait-based ingest pipeline. For opaque files it falls back to filename, path, MIME type, and directory context — which is often sufficient.

Architecture Overview

corten/
├── src/
│   ├── main.rs               # CLI entry point (clap)
│   ├── cli/                  # Subcommand handlers
│   │   ├── init.rs
│   │   ├── tag.rs
│   │   ├── files.rs
│   │   ├── tags.rs
│   │   ├── suggest.rs
│   │   ├── mount.rs
│   │   └── repair.rs
│   ├── db/                   # SQLite layer
│   │   ├── mod.rs
│   │   ├── schema.rs         # DB initialization and migrations
│   │   ├── tags.rs           # Tag + description CRUD
│   │   ├── files.rs          # File registration and lookup
│   │   └── query.rs          # Boolean tag queries
│   ├── ingest/               # File content extraction pipeline
│   │   ├── mod.rs            # Extractor trait definition
│   │   ├── text.rs           # Plaintext, markdown, code, configs
│   │   ├── pdf.rs            # PDF text extraction
│   │   ├── office.rs         # DOCX/XLSX XML extraction
│   │   ├── media.rs          # EXIF, ID3, file metadata
│   │   └── fallback.rs       # Filename + path + MIME fallback
│   ├── ai/                   # Ollama integration
│   │   ├── mod.rs
│   │   ├── client.rs         # HTTP client for Ollama API
│   │   ├── prompt.rs         # Prompt construction
│   │   └── suggest.rs        # Tag suggestion logic
│   ├── vfs/                  # FUSE virtual filesystem
│   │   ├── mod.rs
│   │   ├── mount.rs          # fuse3 mount/unmount
│   │   ├── tags_dir.rs       # /tags/<tagname>/ virtual dirs
│   │   └── queries_dir.rs    # /queries/<expression>/ virtual dirs
│   ├── watcher.rs            # notify crate file watcher
│   └── mcp/                  # MCP server
│       ├── mod.rs
│       ├── server.rs         # MCP server implementation
│       └── tools.rs          # Tool definitions (tag, query, suggest)
├── Cargo.toml
├── justfile
└── README.md

Database Schema

-- Tags with mandatory descriptions
CREATE TABLE tag (
    id          INTEGER PRIMARY KEY,
    name        TEXT NOT NULL UNIQUE,
    description TEXT NOT NULL,          -- REQUIRED, no empty strings
    created_at  DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Files tracked by Corten
CREATE TABLE file (
    id          INTEGER PRIMARY KEY,
    directory   TEXT NOT NULL,
    name        TEXT NOT NULL,
    fingerprint TEXT,                   -- hash for change detection
    created_at  DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(directory, name)
);

-- File <-> Tag relationships
CREATE TABLE file_tag (
    file_id     INTEGER NOT NULL REFERENCES file(id) ON DELETE CASCADE,
    tag_id      INTEGER NOT NULL REFERENCES tag(id) ON DELETE CASCADE,
    source      TEXT NOT NULL DEFAULT 'manual', -- 'manual' | 'ai'
    confidence  REAL,                   -- AI suggestions only, 0.0-1.0
    PRIMARY KEY (file_id, tag_id)
);

-- AI suggestion log (pending review)
CREATE TABLE ai_suggestion (
    id              INTEGER PRIMARY KEY,
    file_id         INTEGER NOT NULL REFERENCES file(id) ON DELETE CASCADE,
    tag_name        TEXT NOT NULL,
    tag_description TEXT NOT NULL,      -- AI must provide this
    confidence      REAL,
    accepted        INTEGER,            -- NULL=pending, 1=accepted, 0=rejected
    created_at      DATETIME DEFAULT CURRENT_TIMESTAMP
);

Key Crates

[dependencies]
# CLI
clap = { version = "4", features = ["derive"] }

# Database
rusqlite = { version = "0.31", features = ["bundled"] }

# FUSE virtual filesystem
fuse3 = "0.7"
tokio = { version = "1", features = ["full"] }

# File watching
notify = "6"

# File content extraction
lopdf = "0.32"          # PDF
zip = "0.6"             # DOCX/XLSX (they are zip archives)
kamadak-exif = "0.5"   # Image EXIF
id3 = "1"               # Audio ID3 tags
infer = "0.15"          # MIME type detection

# AI / Ollama
reqwest = { version = "0.12", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"

# MCP server
axum = "0.7"            # HTTP server for MCP

# Utilities
anyhow = "1"
tracing = "0.1"
tracing-subscriber = "0.3"
dirs = "5"
walkdir = "2"

CLI Interface

corten init                          # Initialize DB at current or home dir
corten tag <file> [files...] <tags>  # Tag one or more files
corten untag <file> <tags>           # Remove tags from a file
corten tags [file]                   # List all tags, or tags on a file
corten files <query>                 # Find files matching tag query
corten suggest <file>                # AI suggests tags for a file
corten suggest --auto <dir>          # AI auto-tags all files in a dir
corten mount <mountpoint>            # Mount FUSE virtual filesystem
corten unmount <mountpoint>          # Unmount
corten repair                        # Resync DB with actual filesystem
corten tag new <name>                # Create a tag (prompts for description)
corten tag describe <name>           # Edit a tag's description

Boolean query syntax (same as TMSU):

corten files "work and important"
corten files "work or personal"
corten files "work and not archived"

AI Auto-tagging — How It Works

Prompt construction

When suggesting tags for a file, Corten:

Extracts a text summary from the file via the ingest pipeline
Loads all existing tags with their descriptions from the DB
Constructs a prompt like:

You are a file tagging assistant. Given a file's content summary and a list of 
existing tags with their descriptions, suggest which tags apply to this file.
You may also suggest new tags if none of the existing ones fit.

IMPORTANT RULES:
- For existing tags: return only tags that genuinely apply based on the description
- For new tags: you MUST provide a description explaining what this tag means
- Return JSON only, no prose

Existing tags:
- "work": Files related to my professional responsibilities at NASA
- "reference": Files I may want to look up later but don't need to act on now
- "personal": Files related to my personal life outside of work

File path: /home/gabriel/Documents/soc-playbook-lateral-movement.pdf
File type: PDF
Content summary: A procedural document describing detection and response steps 
for lateral movement attacks in a SOC environment. Contains MITRE ATT&CK 
references and escalation procedures.

Respond with JSON in this exact format:
{
  "existing_tags": ["work", "reference"],
  "new_tags": [
    {
      "name": "playbook",
      "description": "Documented procedures for responding to specific security events or scenarios"
    }
  ],
  "confidence": 0.92
}

Review flow

High confidence (>0.85) + existing tag: optionally auto-apply (user-configurable threshold)
Lower confidence or new tag: surfaces to user for review via corten suggest --review
User can accept, reject, or edit both the tag name and description before accepting

Ingest Pipeline — Extractor Trait

pub trait Extractor: Send + Sync {
    fn supports(&self, mime: &str, extension: &str) -> bool;
    fn extract(&self, path: &Path) -> anyhow::Result<String>;
}

Extractors are registered in priority order. The pipeline tries each in sequence and falls back to the filename/path extractor if none produce output. Implement a new extractor by implementing this trait and registering it — no other changes needed.

FUSE Virtual Filesystem Layout

<mountpoint>/
├── tags/
│   ├── work/              # symlinks to all files tagged 'work'
│   ├── reference/
│   └── .../
└── queries/
    └── work and reference/   # boolean query as directory name
        └── ...               # symlinks to matching files

This is identical to TMSU's VFS layout so existing TMSU muscle memory transfers directly.

File Watcher

The notify crate watches registered directories for:

New files → register in DB, optionally trigger AI suggestion
Moved files → update path in DB automatically (replaces corten repair)
Deleted files → mark as missing in DB

Run as a background daemon or as a systemd user service:

# ~/.config/systemd/user/corten-watch.service
[Unit]
Description=Corten file watcher

[Service]
ExecStart=/home/gabriel/.local/bin/corten watch
Restart=on-failure

[Install]
WantedBy=default.target

MCP Server

Corten exposes an MCP server so any LLM (LibreChat agents, OpenCode, etc.) can interact with the tag taxonomy directly.

Tools exposed:

corten_tag_file       # Tag a file with one or more tags
corten_query_files    # Find files matching a boolean tag query
corten_list_tags      # List all tags with descriptions
corten_suggest        # Ask Corten to AI-suggest tags for a file
corten_add_tag        # Create a new tag with a description
corten_get_tags       # Get tags on a specific file

Start the MCP server:

corten mcp --port 3456

Ollama Configuration

Corten connects to Ollama at http://ollama.akinus21.com by default.

Config file at ~/.config/corten/config.toml:

[ollama]
base_url = "http://ollama.akinus21.com"
model = "llama3.2"          # or whichever model you prefer
auto_tag_threshold = 0.85   # confidence threshold for auto-apply
max_tags_per_suggest = 10   # max existing tags to include in prompt

[database]
path = "~/.corten/db"       # default DB location

[watcher]
enabled = true
watch_dirs = ["~/Documents", "~/Projects"]

Build Order / Suggested Starting Point

Start with the core data layer before anything else:

corten init + DB schema — get the SQLite schema working with tags requiring descriptions enforced at the DB level
corten tag new — tag creation with interactive description prompt
corten tag <file> — basic file tagging
corten files <query> — boolean query engine
Ingest pipeline — start with text/fallback extractors, add others incrementally
Ollama integration + corten suggest — AI suggestions with description requirement
FUSE VFS — mount/unmount with tags/ and queries/ layout
File watcher — background sync daemon
MCP server — expose tools to LLMs
Corten Beams — cloud layer (separate crate/binary)

Forgejo Repository

Create at: forge.akinus21.com/akinus/corten
Username: akinus
Follow existing CI/CD patterns from akclip for Forgejo Actions pipeline

Notes

This replaces AkTags entirely. AkTags code can be referenced for the SQLite patterns and Iced UI ideas (Corten Patina, future) but the architecture is a clean rewrite.
TMSU's integration test suite hangs on FUSE operations inside containers — do not run FUSE tests in CI, test the VFS layer manually.
The fuse3 crate requires the fuse kernel module. On Bluefin this is available as fuse3 — confirmed present on your setup.

# Corten — Project Guide for OpenCode ## What is Corten? Corten is a Rust-based file tagging system with a FUSE virtual filesystem, semantic tag taxonomy, and built-in AI auto-tagging via Ollama. It is the spiritual successor to AkTags, rebuilt around the TMSU concept but extended with AI integration, a file watcher, and an MCP server. The name comes from corten steel — a material that rusts *intentionally*, forming a protective oxide layer. Corten does the same for your filesystem: it imposes deliberate, protective structure on file chaos. Corten is part of the AK metalworking ecosystem: - **Iron** — browser - **Corten** — file tagger (this project) - **Corten Beams** — cloud/server layer (future) - **Anvil** — global command toolkit (future) --- ## Core Design Philosophy 1. **The tag taxonomy is the memory.** No opaque embeddings. No black box ML. Tag names + human-written descriptions are the only persistent knowledge store. AI decisions are always explainable because they reference visible, editable descriptions. 2. **Every tag has a description.** A tag is not just a word. It is a word + a human-readable statement of purpose. Example: tag `reference`, description `Files I may want to look up later but don't need to act on now.` This description is what the AI reads when deciding whether to apply a tag. 3. **AI suggestions must include descriptions.** When the AI suggests a new tag that doesn't exist yet, it must provide a proposed description. The user sees both the tag name and the AI's reasoning before accepting. 4. **Corrections propagate forward automatically.** When a user edits an AI suggestion or its description, that refined description influences all future tagging decisions without any explicit retraining. The taxonomy gets smarter over time naturally. 5. **Any file type is supported.** Corten extracts meaningful content from any file via a trait-based ingest pipeline. For opaque files it falls back to filename, path, MIME type, and directory context — which is often sufficient. --- ## Architecture Overview ``` corten/ ├── src/ │ ├── main.rs # CLI entry point (clap) │ ├── cli/ # Subcommand handlers │ │ ├── init.rs │ │ ├── tag.rs │ │ ├── files.rs │ │ ├── tags.rs │ │ ├── suggest.rs │ │ ├── mount.rs │ │ └── repair.rs │ ├── db/ # SQLite layer │ │ ├── mod.rs │ │ ├── schema.rs # DB initialization and migrations │ │ ├── tags.rs # Tag + description CRUD │ │ ├── files.rs # File registration and lookup │ │ └── query.rs # Boolean tag queries │ ├── ingest/ # File content extraction pipeline │ │ ├── mod.rs # Extractor trait definition │ │ ├── text.rs # Plaintext, markdown, code, configs │ │ ├── pdf.rs # PDF text extraction │ │ ├── office.rs # DOCX/XLSX XML extraction │ │ ├── media.rs # EXIF, ID3, file metadata │ │ └── fallback.rs # Filename + path + MIME fallback │ ├── ai/ # Ollama integration │ │ ├── mod.rs │ │ ├── client.rs # HTTP client for Ollama API │ │ ├── prompt.rs # Prompt construction │ │ └── suggest.rs # Tag suggestion logic │ ├── vfs/ # FUSE virtual filesystem │ │ ├── mod.rs │ │ ├── mount.rs # fuse3 mount/unmount │ │ ├── tags_dir.rs # /tags/<tagname>/ virtual dirs │ │ └── queries_dir.rs # /queries/<expression>/ virtual dirs │ ├── watcher.rs # notify crate file watcher │ └── mcp/ # MCP server │ ├── mod.rs │ ├── server.rs # MCP server implementation │ └── tools.rs # Tool definitions (tag, query, suggest) ├── Cargo.toml ├── justfile └── README.md ``` --- ## Database Schema ```sql -- Tags with mandatory descriptions CREATE TABLE tag ( id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE, description TEXT NOT NULL, -- REQUIRED, no empty strings created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); -- Files tracked by Corten CREATE TABLE file ( id INTEGER PRIMARY KEY, directory TEXT NOT NULL, name TEXT NOT NULL, fingerprint TEXT, -- hash for change detection created_at DATETIME DEFAULT CURRENT_TIMESTAMP, UNIQUE(directory, name) ); -- File <-> Tag relationships CREATE TABLE file_tag ( file_id INTEGER NOT NULL REFERENCES file(id) ON DELETE CASCADE, tag_id INTEGER NOT NULL REFERENCES tag(id) ON DELETE CASCADE, source TEXT NOT NULL DEFAULT 'manual', -- 'manual' | 'ai' confidence REAL, -- AI suggestions only, 0.0-1.0 PRIMARY KEY (file_id, tag_id) ); -- AI suggestion log (pending review) CREATE TABLE ai_suggestion ( id INTEGER PRIMARY KEY, file_id INTEGER NOT NULL REFERENCES file(id) ON DELETE CASCADE, tag_name TEXT NOT NULL, tag_description TEXT NOT NULL, -- AI must provide this confidence REAL, accepted INTEGER, -- NULL=pending, 1=accepted, 0=rejected created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); ``` --- ## Key Crates ```toml [dependencies] # CLI clap = { version = "4", features = ["derive"] } # Database rusqlite = { version = "0.31", features = ["bundled"] } # FUSE virtual filesystem fuse3 = "0.7" tokio = { version = "1", features = ["full"] } # File watching notify = "6" # File content extraction lopdf = "0.32" # PDF zip = "0.6" # DOCX/XLSX (they are zip archives) kamadak-exif = "0.5" # Image EXIF id3 = "1" # Audio ID3 tags infer = "0.15" # MIME type detection # AI / Ollama reqwest = { version = "0.12", features = ["json"] } serde = { version = "1", features = ["derive"] } serde_json = "1" # MCP server axum = "0.7" # HTTP server for MCP # Utilities anyhow = "1" tracing = "0.1" tracing-subscriber = "0.3" dirs = "5" walkdir = "2" ``` --- ## CLI Interface ``` corten init # Initialize DB at current or home dir corten tag <file> [files...] <tags> # Tag one or more files corten untag <file> <tags> # Remove tags from a file corten tags [file] # List all tags, or tags on a file corten files <query> # Find files matching tag query corten suggest <file> # AI suggests tags for a file corten suggest --auto <dir> # AI auto-tags all files in a dir corten mount <mountpoint> # Mount FUSE virtual filesystem corten unmount <mountpoint> # Unmount corten repair # Resync DB with actual filesystem corten tag new <name> # Create a tag (prompts for description) corten tag describe <name> # Edit a tag's description ``` ### Boolean query syntax (same as TMSU): ``` corten files "work and important" corten files "work or personal" corten files "work and not archived" ``` --- ## AI Auto-tagging — How It Works ### Prompt construction When suggesting tags for a file, Corten: 1. Extracts a text summary from the file via the ingest pipeline 2. Loads all existing tags with their descriptions from the DB 3. Constructs a prompt like: ``` You are a file tagging assistant. Given a file's content summary and a list of existing tags with their descriptions, suggest which tags apply to this file. You may also suggest new tags if none of the existing ones fit. IMPORTANT RULES: - For existing tags: return only tags that genuinely apply based on the description - For new tags: you MUST provide a description explaining what this tag means - Return JSON only, no prose Existing tags: - "work": Files related to my professional responsibilities at NASA - "reference": Files I may want to look up later but don't need to act on now - "personal": Files related to my personal life outside of work File path: /home/gabriel/Documents/soc-playbook-lateral-movement.pdf File type: PDF Content summary: A procedural document describing detection and response steps for lateral movement attacks in a SOC environment. Contains MITRE ATT&CK references and escalation procedures. Respond with JSON in this exact format: { "existing_tags": ["work", "reference"], "new_tags": [ { "name": "playbook", "description": "Documented procedures for responding to specific security events or scenarios" } ], "confidence": 0.92 } ``` ### Review flow - **High confidence (>0.85) + existing tag**: optionally auto-apply (user-configurable threshold) - **Lower confidence or new tag**: surfaces to user for review via `corten suggest --review` - User can accept, reject, or edit both the tag name and description before accepting --- ## Ingest Pipeline — Extractor Trait ```rust pub trait Extractor: Send + Sync { fn supports(&self, mime: &str, extension: &str) -> bool; fn extract(&self, path: &Path) -> anyhow::Result<String>; } ``` Extractors are registered in priority order. The pipeline tries each in sequence and falls back to the filename/path extractor if none produce output. Implement a new extractor by implementing this trait and registering it — no other changes needed. --- ## FUSE Virtual Filesystem Layout ``` <mountpoint>/ ├── tags/ │ ├── work/ # symlinks to all files tagged 'work' │ ├── reference/ │ └── .../ └── queries/ └── work and reference/ # boolean query as directory name └── ... # symlinks to matching files ``` This is identical to TMSU's VFS layout so existing TMSU muscle memory transfers directly. --- ## File Watcher The `notify` crate watches registered directories for: - New files → register in DB, optionally trigger AI suggestion - Moved files → update path in DB automatically (replaces `corten repair`) - Deleted files → mark as missing in DB Run as a background daemon or as a systemd user service: ```ini # ~/.config/systemd/user/corten-watch.service [Unit] Description=Corten file watcher [Service] ExecStart=/home/gabriel/.local/bin/corten watch Restart=on-failure [Install] WantedBy=default.target ``` --- ## MCP Server Corten exposes an MCP server so any LLM (LibreChat agents, OpenCode, etc.) can interact with the tag taxonomy directly. ### Tools exposed: ``` corten_tag_file # Tag a file with one or more tags corten_query_files # Find files matching a boolean tag query corten_list_tags # List all tags with descriptions corten_suggest # Ask Corten to AI-suggest tags for a file corten_add_tag # Create a new tag with a description corten_get_tags # Get tags on a specific file ``` Start the MCP server: ```bash corten mcp --port 3456 ``` --- ## Ollama Configuration Corten connects to Ollama at `http://ollama.akinus21.com` by default. Config file at `~/.config/corten/config.toml`: ```toml [ollama] base_url = "http://ollama.akinus21.com" model = "llama3.2" # or whichever model you prefer auto_tag_threshold = 0.85 # confidence threshold for auto-apply max_tags_per_suggest = 10 # max existing tags to include in prompt [database] path = "~/.corten/db" # default DB location [watcher] enabled = true watch_dirs = ["~/Documents", "~/Projects"] ``` --- ## Build Order / Suggested Starting Point Start with the core data layer before anything else: 1. **`corten init` + DB schema** — get the SQLite schema working with tags requiring descriptions enforced at the DB level 2. **`corten tag new`** — tag creation with interactive description prompt 3. **`corten tag <file>`** — basic file tagging 4. **`corten files <query>`** — boolean query engine 5. **Ingest pipeline** — start with text/fallback extractors, add others incrementally 6. **Ollama integration + `corten suggest`** — AI suggestions with description requirement 7. **FUSE VFS** — mount/unmount with tags/ and queries/ layout 8. **File watcher** — background sync daemon 9. **MCP server** — expose tools to LLMs 10. **Corten Beams** — cloud layer (separate crate/binary) --- ## Forgejo Repository - Create at: `forge.akinus21.com/akinus/corten` - Username: `akinus` - Follow existing CI/CD patterns from `akclip` for Forgejo Actions pipeline --- ## Notes - This replaces AkTags entirely. AkTags code can be referenced for the SQLite patterns and Iced UI ideas (Corten Patina, future) but the architecture is a clean rewrite. - TMSU's integration test suite hangs on FUSE operations inside containers — do not run FUSE tests in CI, test the VFS layer manually. - The `fuse3` crate requires the `fuse` kernel module. On Bluefin this is available as `fuse3` — confirmed present on your setup.

akinus added the

enhancement

label

2026-06-28 03:19:43 +00:00