# QSV Agent Skills

Complete TypeScript implementation for loading, executing, and composing qsv command pipelines with the Claude Agent SDK and Claude Desktop (MCP).

## 🎯 NEW: Work with Local Tabular Data Files Without Uploading!

The QSV MCP Server now supports **direct access to local tabular data files** (CSV, Excel, JSONL, etc.). No more uploading files to Claude Desktop!

**Getting Started**: See [docs/guides/START_HERE.md](./docs/guides/START_HERE.md)
**Full Guide**: See [docs/guides/FILESYSTEM_USAGE.md](./docs/guides/FILESYSTEM_USAGE.md)

### Key Features:
- ✅ Browse tabular data files in your directories (CSV, Excel, JSONL, etc.)
- ✅ Process files without uploading
- ✅ No input file size limits (streams large files efficiently)
- ✅ Smart output handling (auto-saves large results > 850KB to disk)
- ✅ Instant access
- ✅ Secure path validation

## Overview

This directory contains:

1. **55 Auto-generated Skill Definitions** - JSON files describing all qsv commands (parsed with qsv-docopt)
2. **TypeScript Executor** - Complete implementation for running qsv skills
3. **MCP Server with Filesystem Access** - Model Context Protocol server for Claude Desktop integration
4. **Working Demos** - Practical demonstrations of the system

Each skill file provides:
- **Command specification**: Binary, subcommand, arguments, and options (parsed with qsv-docopt)
- **Rich descriptions**: Extracted from usage text
- **Usage examples**: Real usage examples from documentation (221 total)
- **Type information**: Inferred parameter types and validation
- **Performance hints**: Memory usage, streaming capability, indexing benefits
- **Links to tests**: For additional context and validation

## Quick Start

### Installation

**Choose your installation method:**

<details>
<summary><b>🎁 Desktop Extension (Recommended for GUI users)</b> - Easiest installation, automatic updates, zero configuration</summary>

**Perfect for**: Most users, especially non-technical users

**Requirements**: Claude Desktop + qsv binary

**Installation**:
1. Install qsv: `brew install qsv` (macOS) or see [qsv installation](https://github.com/dathere/qsv#installation)
2. Download `qsv-mcp-server.mcpb` from releases
3. Double-click the `.mcpb` file (opens with Claude Desktop)
4. **That's it!** qsv binary path is auto-detected, no manual configuration needed
5. Restart Claude Desktop

Auto-detection finds qsv in standard locations (PATH, /usr/local/bin, ~/.cargo/bin, etc.)

**Documentation**: [docs/guides/START_HERE.md](./docs/guides/START_HERE.md)

</details>

<details>
<summary><b>💻 Claude Code (Recommended for CLI users)</b> - Terminal-based, perfect for developers</summary>

**Perfect for**: Developers, terminal users, automation, remote servers, SSH workflows

**Requirements**: Claude Code CLI + Node.js ≥ 20.0.0 + qsv binary

**Installation**:
```bash
cd .claude/skills
npm install
npm run build
npm run mcp:install  # Detects and configures Claude Code automatically
```

**Benefits**:
- ✅ Terminal integration - work with files in current directory
- ✅ Scriptable - automate data workflows
- ✅ Works over SSH - process data on remote servers
- ✅ Git integration - see operations in repository context
- ✅ Lightweight - no GUI overhead

**Documentation**: [docs/guides/CLAUDE_CODE.md](./docs/guides/CLAUDE_CODE.md)

</details>

<details>
<summary><b>⚙️ Legacy MCP Server</b> - Maximum flexibility, for advanced users</summary>

**Perfect for**: Advanced users who want full config file control, using non-Claude MCP clients

**Requirements**: Node.js ≥ 20.0.0 + npm + qsv binary

**Installation**:
```bash
cd .claude/skills
npm install
npm run build
npm run mcp:install  # Interactive configuration
```

This will:
1. Build TypeScript code
2. Prompt for configuration (qsv path, allowed directories, etc.)
3. Update Claude Desktop config file automatically
4. Restart Claude Desktop to activate

**Manual configuration**: Edit `~/Library/Application Support/Claude/claude_desktop_config.json`

</details>

**All methods provide identical qsv functionality** - choose based on your interface preference (GUI vs CLI) and use case.

### Run Examples (Legacy MCP Server only)

```bash
# Basic skill loading and execution
npm test

# Package as Desktop Extension
npm run mcpb:package
```

## Generated Skills (55)

| Category | Count | Skills |
|----------|-------|--------|
| **utility** | 24 | blake3, cat, dedup, diff, enum, exclude, explode, extdedup, extsort, fill, geocode, headers, implode, index, luau, partition, pivotp, pseudo, sniff, sort, sortcheck, split, sqlp, template |
| **conversion** | 6 | excel, input, json, jsonl, to, tojsonl |
| **aggregation** | 5 | count, frequency, moarstats, pragmastat, stats |
| **transformation** | 5 | datefmt, rename, replace, reverse, transpose |
| **selection** | 3 | sample, select, slice |
| **formatting** | 3 | fixlengths, fmt, table |
| **validation** | 3 | safenames, schema, validate |
| **filtering** | 2 | search, searchset |
| **joining** | 2 | join, joinp |
| **documentation** | 1 | describegpt |
| **generation** | 1 | synthesize |

**Total Statistics:**
- **Skills**: 55 commands
- **Usage Examples**: 221 from documentation
- **Options**: 658 command-line options
- **Arguments**: 98 positional arguments

## Project Structure

```
.claude/skills/
├── qsv/                    # 55 skill JSON definitions
│   ├── qsv-select.json
│   ├── qsv-stats.json
│   ├── qsv-moarstats.json
│   └── ... (52 more)
├── src/                    # TypeScript source
│   ├── types.ts           # Type definitions
│   ├── loader.ts          # Skill loading
│   ├── executor.ts        # qsv execution wrapper
│   ├── mcp-server.ts      # MCP server implementation
│   ├── mcp-tools.ts       # MCP tool definitions
│   ├── mcp-filesystem.ts  # Filesystem resource provider
│   ├── mcp-sampling.ts    # MCP sampling support
│   ├── bm25-search.ts     # BM25 search index for tool discovery
│   ├── browse-directory.ts # Directory browser (MCP Apps)
│   ├── config.ts          # Configuration and validation
│   ├── converted-file-manager.ts  # LIFO cache for converted files
│   ├── duckdb.ts          # DuckDB integration for SQL queries
│   ├── installer.ts       # Binary installer
│   ├── update-checker.ts  # Version detection and skill regeneration
│   ├── utils.ts           # Utility functions
│   ├── version.ts         # Version management
│   ├── wink-bm25-text-search.d.ts  # BM25 type declarations
│   ├── wink-nlp-utils.d.ts         # NLP utils type declarations
│   ├── ui/                # UI components
│   │   └── directory-picker-html.ts
│   └── index.ts           # Public exports
├── scripts/
│   ├── install-mcp.js     # MCP installation helper
│   ├── package-mcpb.js    # MCPB packaging script
│   ├── package-plugin.js  # Plugin packaging script
│   ├── cowork-setup.cjs   # Claude Cowork integration setup
│   └── run-tests.js       # Cross-platform test runner
├── dist/                   # Compiled JavaScript (gitignored)
├── package.json
├── tsconfig.json
├── mcp-config.json         # Claude Desktop config template
├── README.md               # This file
├── README-MCP.md           # MCP server documentation
├── CHANGELOG.md            # Release notes
└── docs/                   # Documentation
    ├── guides/             # User guides
    ├── reference/          # Technical reference
    └── desktop/            # Desktop extension documentation
```

## Usage

### Basic Skill Execution

```typescript
import { SkillLoader, SkillExecutor } from './dist/index.js';

// Load all 55 skills
const loader = new SkillLoader();
await loader.loadAll();

// Search for skills
const dedupSkills = loader.search('duplicate');
// Returns: qsv-dedup, qsv-extdedup, qsv-diff, etc.

// Execute a skill
const executor = new SkillExecutor();
const skill = await loader.load('qsv-select');

const result = await executor.execute(skill, {
  args: { selection: '1-5' },
  stdin: csvData
});

console.log(result.output);
```

## Skill Schema

Each skill JSON file follows this structure:

```json
{
  "name": "qsv-<command>",
  "version": "12.0.0",
  "description": "Command description from usage text",
  "category": "selection|filtering|transformation|aggregation|...",
  "command": {
    "binary": "qsv",
    "subcommand": "<command>",
    "args": [
      {
        "name": "argument_name",
        "type": "string|number|file|regex",
        "required": true|false,
        "description": "Argument description",
        "examples": []
      }
    ],
    "options": [
      {
        "flag": "--option",
        "short": "-o",
        "type": "flag|string|number",
        "description": "Option description",
        "default": "value"
      }
    ]
  },
  "examples": [
    {
      "description": "What this example does",
      "command": "qsv command example"
    }
  ],
  "hints": {
    "streamable": true,
    "indexed": false,
    "memory": "constant"
  },
  "test_file": "https://github.com/dathere/qsv/blob/master/tests/test_<command>.rs"
}
```

## Generation

### Skill Definitions

Skills are auto-generated from qsv command USAGE text using the `--update-mcp-skills` flag:

```bash
# Must be run from within the qsv repository directory
cd /path/to/qsv
qsv --update-mcp-skills

# Output: .claude/skills/qsv/*.json
```

The generator uses **qsv-docopt Parser** (the same parser qsv uses at runtime) for robust parsing:
1. Extracts `USAGE` static string from command source files
2. **Parses with qsv-docopt** for accurate argument/option detection
3. Extracts descriptions from USAGE text
4. Infers types from names and descriptions
5. Detects performance hints from emoji markers (🤯 📇 🏎️ 😣)
6. Generates structured JSON skill definitions

## Type Inference

The generator infers parameter types:

| Pattern | Inferred Type |
|---------|---------------|
| `<input>`, `<file>` | `file` |
| `<number>`, `<count>` | `number` |
| `<regex>`, `<pattern>` | `regex` |
| `<selection>`, `<column>` | `string` |
| Default | `string` |

## Performance Hints

Hints are extracted from usage text markers:

- 🤯 → `memory: "full"` (loads entire file)
- 📇 → `indexed: true` (benefits from index)
- 🏎️ → `parallel: true` (supports parallel execution)
- 😣 → `memory: "proportional"` (memory scales with cardinality)

## API Documentation

See [docs/reference/SKILLS_API.md](./docs/reference/SKILLS_API.md) for complete API documentation including:

- SkillLoader API (loading, searching, statistics)
- SkillExecutor API (execution, validation)
- QsvPipeline API (15 convenience methods)
- Type definitions
- Advanced examples

## Integration with Claude Agent SDK

```typescript
import { Agent } from '@anthropic-ai/agent-sdk';
import { SkillLoader } from './dist/index.js';

const loader = new SkillLoader();
await loader.loadAll();

const agent = new Agent({
  skills: Array.from((await loader.loadAll()).values())
});

// Agent can now discover and invoke qsv skills
await agent.chat("Remove duplicates from sales.csv");
// Agent automatically finds and invokes qsv-dedup
```

## Integration with Claude Desktop (MCP Server)

The QSV MCP Server exposes all 55 qsv skill-based commands to Claude Desktop through the Model Context Protocol.

### Quick Start

```bash
# Install and configure
cd .claude/skills
npm install
npm run mcp:install
```

This will:
1. Build the MCP server
2. Update Claude Desktop configuration
3. Enable qsv tools in Claude Desktop

### Manual Configuration

Edit `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "qsv": {
      "command": "node",
      "args": ["/path/to/qsv/.claude/skills/dist/mcp-server.js"],
      "env": {
        "QSV_MCP_BIN_PATH": "/usr/local/bin/qsv",
        "QSV_MCP_WORKING_DIR": "/Users/your-username/Downloads",
        "QSV_MCP_ALLOWED_DIRS": "/Users/your-username/Downloads:/Users/your-username/Documents"
      }
    }
  }
}
```

Restart Claude Desktop to load the server.

### Environment Variables

The `env` block above isn't limited to `QSV_MCP_*` keys — **any `QSV_*` variable qsv understands is honored**. The MCP server spawns the qsv binary as a child process and forwards its full `process.env` unchanged, so vars set in `env` reach qsv directly.

Common reasons to set additional vars here:

- **Geocoding API key**: `QSV_OPENCAGE_API_KEY`
- **Geocoding cache**: `QSV_GEOCODE_INDEX_FILENAME`, `QSV_CACHE_DIR`
- **CKAN integration**: `QSV_CKAN_API`, `QSV_CKAN_TOKEN`
- **Reader/writer buffers**: `QSV_RDR_BUFFER_CAPACITY`, `QSV_WTR_BUFFER_CAPACITY`

Two caveats:

- The MCPB `user_config` UI in Claude Desktop only exposes the keys the manifest declares; any non-`QSV_MCP_*` var must be added by hand to the `env` block (or exported in the shell that launches the MCP client).
- Some qsv env vars are read once at startup (e.g., update-check flags), so changes take effect only after restarting the MCP server.

For the full catalog — both general qsv env vars and the dedicated `## MCP Server Environment Variables` section — see [docs/ENVIRONMENT_VARIABLES.md](../../docs/ENVIRONMENT_VARIABLES.md).

### Usage

Once configured, use natural language in Claude Desktop:

```
"Select columns 1-5 from data.csv"
"Calculate statistics for the price column in sales.csv"
"Remove duplicates from data.csv and sort by revenue"
"Show me an example of joining two CSV files"
```

Claude will automatically:
- Select the appropriate qsv tool
- Execute the command
- Return results or explanations

### What's Available

- **24+ MCP Tools**: 10 core tools (search, config, set_working_dir, get_working_dir, filesystem, log, command, to_parquet, index, stats) + 13 common command tools (including sniff). `qsv_browse_directory` is also available when MCP Apps are supported.
- **Local File Access**: Browse and process tabular data files (CSV, Excel, JSONL, etc.) directly from your filesystem
- **File-Based Processing**: Works with your local files without uploading
- **Natural Language Interface**: No command syntax needed

For complete MCP documentation, see [README-MCP.md](./README-MCP.md).

## Development

```bash
# Install dependencies
npm install

# Build TypeScript
npm run build

# Run demos
npm test                              # Basic skill usage
npm run mcp:install                   # Install MCP server for Claude Desktop

# Regenerate skills (from qsv repository root)
cd /path/to/qsv
qsv --update-mcp-skills
```

## Documentation

### Installation & Setup
- [Getting Started Guide](./docs/guides/START_HERE.md) - Install qsv MCP Server + Cowork Plugin (recommended)
- [MCP Server Guide](./README-MCP.md) - Claude Desktop integration (legacy method)
- [Auto-Update Guide](./docs/reference/AUTO_UPDATE.md) - Keep skills in sync with qsv releases
- [Environment Variables](../../docs/ENVIRONMENT_VARIABLES.md) - Full catalog of `QSV_*` vars and MCP-specific knobs

### Usage & Features
- [Additional Servers Guide](./docs/guides/ADDITIONAL_SERVERS.md) - Census + Wikidata MCP servers
- [Filesystem Usage Guide](./docs/guides/FILESYSTEM_USAGE.md) - Local file access
- [Complete API Documentation](./docs/reference/SKILLS_API.md)
- [qsv Commands](https://github.com/dathere/qsv#commands)

## Requirements

- Node.js >= 20.0.0
- qsv installed and in PATH (for execution)
- TypeScript 5.0+ (for development)

## License

MIT

---

**Updated**: 2026-05-18
**Version**: 20.1.0
**Generator**: `qsv --update-mcp-skills`
**Skills**: 55 commands
**Usage Examples**: 221 from documentation
**Parsing**: qsv-docopt (robust, accurate)
**Features**: MCP server, filesystem access, type-safe execution
**Status**: ✅ Production Ready
