ATAP AI Annotator

AI Text Classification Platform

Version 0.1.0

📁 View on GitHub

🔒 Private data supported: Process sensitive data locally with Ollama (no data leaves your machine)
100+ LLM Providers via LiteLLM
3 Reasoning Features
7 REST API Endpoints
3 Prompting Techniques
5 Concurrent Workers
125 Comprehensive Tests
⌨️

Command Line Interface

Powerful CLI for batch processing and automation

  • Batch classification with progress tracking
  • CSV/XLSX dataset support
  • Multiple output formats (JSON, CSV, ZIP)
  • 🔒 Local model support via Ollama for private data
  • Confidence scores and reasoning output
  • Native reasoning mode support (o1, o3-mini)
  • Model listing and discovery
  • Cost estimation before processing
  • Rich console output with status updates
🌐

REST API (FastAPI)

Modern async API for seamless integration

  • 7 comprehensive endpoints
  • Job management with UUID tracking
  • Real-time progress monitoring
  • Confidence scores and reasoning output
  • Native reasoning mode support
  • Cost estimation endpoint
  • Interactive Swagger UI documentation
  • API key authentication
🖥️

Web App (Streamlit)

Interactive UI for easy classification tasks

  • User-friendly web interface
  • CSV file upload with column selection
  • Schema templates and JSON editor
  • Real-time progress monitoring
  • Confidence scores and reasoning output
  • Native reasoning mode configuration
  • Cost estimation before processing
  • Results download as CSV
  • Built on top of FastAPI backend

Classification Techniques Explained

Understanding the three prompting techniques with schema examples

Zero-Shot Classification

Direct classification without any examples. The model uses only class descriptions to make decisions.

User Schema:
{ "classes": [ { "name": "positive", "description": "Text expressing positive sentiment" }, { "name": "negative", "description": "Text expressing negative sentiment" }, { "name": "neutral", "description": "Text with neutral or no sentiment" } ] }
Best for: Quick classification, instruction-following models, clear-cut categories

Few-Shot Classification

Classification with demonstration examples. The model learns from provided query-classification pairs.

User Schema:
{ "classes": [ { "name": "positive", "description": "Positive sentiment" }, { "name": "negative", "description": "Negative sentiment" } ], "examples": [ { "query": "This product exceeded my expectations!", "classification": "positive" }, { "query": "Terrible service, very disappointed.", "classification": "negative" } ] }
Best for: Complex classifications, nuanced categories, improving accuracy with demonstrations

Chain-of-Thought Classification

Reasoning-based classification. The model explains its thinking process before making a decision.

User Schema:
{ "classes": [ { "name": "urgent", "description": "Requires immediate action" }, { "name": "normal", "description": "Standard priority" } ], "examples": [ { "query": "Server is down, clients can't access the system", "classification": "urgent", "reason": "Service outage affecting multiple clients requires immediate attention" }, { "query": "Update documentation for the new feature", "classification": "normal", "reason": "Documentation update is important but not time-sensitive" } ] }
Best for: Complex reasoning tasks, interpretable classifications, high-stakes decisions requiring explanation

Reasoning & Confidence Features

Enhanced classification with confidence scores, reasoning explanations, and native reasoning modes

Confidence Scores

Every classification automatically includes a confidence score (0.0 to 1.0) indicating the model's certainty.

Example Output:
{ "classification": "positive", "confidence": 0.95, "text": "This product is amazing!" }
Always included: No configuration needed. Available in all results.

Reasoning Output

Optional explanations from the LLM about why it made each classification decision. Enable with --enable-reasoning flag.

Example Output:
{ "classification": "positive", "confidence": 0.92, "reasoning": "The text uses strong positive language like 'amazing' and 'excellent', indicating high satisfaction." }
Best for: Understanding model decisions, debugging misclassifications, providing explanations to end users

Native Reasoning Modes

For reasoning-capable models, enable deeper analysis with --reasoning-effort (low/medium/high).

Example Output:
{ "classification": "urgent", "confidence": 0.98, "reasoning_content": "After analyzing the context, the mention of 'server down' and 'clients can't access' indicates immediate business impact requiring urgent attention..." }
Best for: Complex tasks requiring deep analysis, high-stakes decisions, maximum accuracy

Feature Comparison

Feature Always Included Parameter Output Field
Confidence Score ✓ Yes - confidence
Reasoning Output Optional --enable-reasoning reasoning
Native Reasoning Optional --reasoning-effort reasoning_content

Powered by LiteLLM

The ATAP AI Annotator leverages LiteLLM's unified API to provide seamless access to multiple LLM providers

100+ LLM Providers Supported

OpenAI • Anthropic • Google • AWS Bedrock • Azure • Cohere • Ollama (Local) • HuggingFace • and many more

One API interface • Automatic cost tracking • Token counting • Dynamic model discovery

🔒 Privacy-First Option: Use Ollama for completely local processing of sensitive data

REST API Endpoints

Health & Status

GET /health Service health check with uptime
GET /health/ready Readiness probe for orchestration

Classification Operations

POST /classify/batch Submit batch classification job with async processing
POST /classify/estimate-cost Estimate tokens and USD cost before processing

Job Management

GET /jobs/{job_id} Get job status, progress, results, and cost
DELETE /jobs/{job_id} Cancel running or pending job

Model Information

GET /models List all available models with pricing and context windows

🧠Prompting Techniques

  • Zero-Shot: Direct classification without examples
  • Few-Shot: Learn from demonstration examples
  • Chain-of-Thought: Reasoning-based classification

🔌LLM Providers (via LiteLLM)

  • 100+ Providers Supported through LiteLLM unified API
  • OpenAI: GPT-4, GPT-3.5, fine-tuned models
  • Ollama: 🔒 Local models for private/sensitive data
  • Azure OpenAI: Enterprise deployment
  • Anthropic, Google, AWS Bedrock, Cohere, and more
  • Custom endpoint support
  • Dynamic model discovery from LiteLLM database

⚙️Processing Features

  • Async batch processing
  • 5 concurrent workers by default
  • Token bucket rate limiting
  • Exponential backoff retry logic
  • Progress tracking and callbacks
  • Max 1000 texts per batch

💰Cost Management

  • Pre-job cost estimation
  • Real-time token counting
  • Per-model pricing lookup
  • Input/output token breakdown
  • Total USD cost calculation
  • LiteLLM pricing integration

🔒Security & Auth

  • API key header authentication
  • Per-request LLM API key override
  • Environment-based configuration
  • Secure credential handling
  • CORS middleware support

📊Job Management

  • 5 job states: PENDING, RUNNING, COMPLETED, FAILED, CANCELLED
  • UUID-based job tracking
  • Progress percentage calculation
  • Error logging per item
  • Max 100 concurrent jobs
  • 3600s default timeout

📝Output & Validation

  • JSON structured output format (YAML also supported)
  • Pydantic model validation
  • User schema validation
  • Comprehensive error messages
  • Multiple output file formats

🧠Reasoning & Confidence

  • Confidence Scores: Automatic confidence metrics (0-1) for every classification
  • Reasoning Output: Optional explanations for classification decisions
  • Native Reasoning: Support for o1/o3-mini models with configurable effort levels
  • Configurable reasoning length limits
  • Available across CLI, Python API, and REST API

🖥️Web App Features

  • 3 pre-built schema templates
  • Interactive JSON schema editor
  • Auto-refresh job polling
  • Visual progress tracking with metrics
  • CSV preview and column selector
  • One-click job cancel and refresh
  • Results download as CSV
  • Confidence scores and reasoning output support
  • Native reasoning mode configuration
  • No coding required

System Architecture

Input Layer

CLI / REST API / WebApp

CSV, XLSX, JSON

→

Pipeline Engine

Async Processing

Rate Limiting

→

LiteLLM Router

100+ LLM Providers

Unified API Interface

→

Results

JSON, CSV

Cost Reports

Technology Stack

Python 3.11+
Core Language
FastAPI
REST API Framework
Typer
CLI Framework
Streamlit
Web App Framework
LiteLLM
Unified LLM API (100+ Providers)
Ollama
Local LLM Runtime
Pydantic
Data Validation
AsyncIO
Concurrency
Poetry
Dependency Management
Pytest
Testing Framework
Loguru
Logging
Ruff
Linting & Formatting

Built for LDaCA/ATAP Platform

Research-grade AI text classification with flexibility, reliability, and transparency