SDK Documentation

Build Voice AI Agents
That Feel Human

The Ferni SDK provides everything you need to create voice AI agents with personality, emotion intelligence, and natural conversation flow.

Installation

Install the Ferni SDK via npm or pnpm:

bash 
# Using npm
npm install @ferni/agents

# Using pnpm (recommended - 4x faster)
pnpm add @ferni/agents

Quick Start

Create your first voice AI agent in minutes:

typescript 
import { createVoiceAgent } from '@ferni/agents';

const agent = await createVoiceAgent({
  personaId: 'my-coach',
  voice: {
    provider: 'cartesia',
    voiceId: 'a0e99841-438c-4a64-b679-ae501e7d6091',
  },
  personality: {
    warmth: 0.85,
    humorLevel: 0.6,
    formality: 0.3,
  },
});

// Start listening
await agent.start();
console.log('Agent is ready to talk!');

Persona Bundle Structure

Personas are the heart of Ferni agents. Each persona bundle contains identity, behaviors, and knowledge:

directory structure 
bundles/my-coach/
├── persona.manifest.json      # Configuration & metadata
├── identity/
│   ├── system-prompt.md       # Core personality prompt
│   ├── biography.md           # Background story
│   └── directors-notes.md     # Behavioral guidelines
├── content/
│   ├── behaviors/
│   │   ├── greetings.json     # Opening lines & greetings
│   │   ├── emotional-intelligence.json
│   │   ├── backchannels.json  # Active listening ("mm-hmm")
│   │   └── cognitive.json     # Reasoning style
│   ├── voice/
│   │   └── expressions.json   # Voice expression patterns
│   ├── stories/
│   │   └── anecdotes.json     # Personal stories to share
│   └── knowledge/
│       └── expertise.json     # Domain knowledge
└── index.ts                   # Entry point
1

persona.manifest.json Schema

The manifest defines your agent's core configuration:

persona.manifest.json 
{
  "identity": {
    "id": "my-coach",
    "name": "Luna",
    "display_name": "Luna the Life Coach",
    "description": "Your thoughtful companion for growth",
    "self_reference": "Luna"
  },

  "voice": {
    "provider": "cartesia",
    "voice_id": "a0e99841-438c-4a64-b679-ae501e7d6091",
    "default_rate": "medium"
  },

  "personality": {
    "warmth": 0.85,
    "humor_level": 0.6,
    "formality": 0.3,
    "energy": 0.8,
    "traits": ["empathetic", "encouraging", "insightful"]
  },

  "cognitive": {
    "reasoning_style": "empathetic",
    "uncertainty_response": "synthesize",
    "attention_focus": ["emotions", "relationships", "growth"],
    "adaptiveness": 0.85
  },

  "role": {
    "id": "life-coach",
    "domains": ["life-coaching", "goal-setting", "personal-growth"],
    "can_handoff": true
  }
}
2

Voice Configuration

Configure voice synthesis with Cartesia:

typescript 
// In persona.manifest.json
{
  "voice": {
    "provider": "cartesia",
    "voice_id": "a0e99841-438c-4a64-b679-ae501e7d6091",
    "default_rate": "medium"  // slow, medium, fast
  },

  "speech_characteristics": {
    "base_speed_multiplier": 0.85,  // 0.5-2.0
    "pause_multiplier": 1.0,         // Adjust pauses
    "thinking_sound_frequency": 0.45, // "um", "uh" frequency
    "emphasis_style": "moderate"      // subtle, moderate, dramatic
  },

  "emotional": {
    "voice_expression": {
      "mirroring_level": 0.6,        // Mirror user's emotion
      "default_tone": "warm",
      "contextual_tones": {
        "distressed_user": "compassionate",
        "celebrating_user": "warm",
        "confused_user": "patient",
        "angry_user": "calm"
      }
    }
  }
}

Browse available voices at cartesia.ai/voices

3

Personality Traits

Define your agent's personality with numeric scales and traits:

json 
{
  "personality": {
    "warmth": 0.9,          // 0.0 (cold) - 1.0 (very warm)
    "humor_level": 0.8,     // 0.0 (serious) - 1.0 (playful)
    "formality": 0.3,       // 0.0 (casual) - 1.0 (formal)
    "directness": 0.7,      // 0.0 (indirect) - 1.0 (blunt)
    "energy": 0.85,         // 0.0 (calm) - 1.0 (energetic)

    "humor_style": [
      "dry_wit",
      "self_deprecating",
      "observational"
    ],

    "traits": [
      "empathetic",
      "curious",
      "patient",
      "encouraging",
      "insightful"
    ]
  }
}
4

Agent Handoff System

Enable seamless handoffs between specialized agents:

json 
{
  "role": {
    "id": "life-coach",
    "domains": ["life-coaching", "personal-growth"],
    "can_handoff": true,
    "handoff_targets": ["@team"]  // Can handoff to any team member
  },

  "handoff": {
    "transition": {
      "style": "warm",         // warm | standard | dramatic | subtle
      "emoji": "",             // Empty string = auto-derive
      "sound": "connect",      // Transition sound effect
      "delay_multiplier": 1.0
    },

    "entrance_phrases": [
      "Hey! I'm here now. What can we explore together?",
      "Back with you! What's next?"
    ],

    "triggers": [
      "talk to coach",
      "life coaching",
      "need guidance"
    ]
  },

  "team": {
    "membership": "ferni-team",
    "coordinator": true,         // This agent coordinates handoffs
    "handoff_triggers": [
      "hey coach",
      "talk to luna"
    ]
  }
}
5

Tool Integration

Give your agent access to tools organized by domain:

json 
{
  "tools": {
    "domains": [
      "memory",           // Remember user preferences
      "information",      // Web search, news, weather
      "entertainment",    // Music, games
      "awareness",        // Mindfulness, presence
      "relationships",    // Connection, vulnerability
      "simple-utilities"  // Time, date, timers
    ],

    "required": [
      "rememberAboutMe",
      "whatDoYouKnowAboutMe"
    ],

    "optional": [
      "searchWeb",
      "getWeather",
      "playMusic"
    ],

    "forbidden": [
      "medicalDiagnosis",  // Out of scope
      "legalAdvice"
    ]
  }
}

See Tool Domains Reference for all available tools.

6

Emotional Intelligence

Enable "Better than Human" emotional intelligence features:

json 
{
  "emotional": {
    "emotion_detection": {
      "enabled": true,
      "sensitivity": "high",      // low, medium, high
      "response_delay_ms": 400
    },

    "voice_expression": {
      "mirroring_level": 0.6,     // Mirror user's emotion (0-1)
      "default_tone": "warm",
      "contextual_tones": {
        "distressed_user": "compassionate",
        "celebrating_user": "warm",
        "confused_user": "patient"
      }
    },

    "empathy": {
      "acknowledgment_frequency": "always",
      "validation_style": "warm",
      "comfort_phrases": [
        "I hear you. That sounds really hard.",
        "It's okay to feel this way."
      ]
    }
  },

  "humanization": {
    "preset": "therapeutic",      // casual | professional | therapeutic

    "overrides": {
      "disfluency": {
        "enabled": true,
        "frequency": 0.12         // "um", "uh" frequency
      },

      "active_listening": {
        "enabled": true,
        "backchannel_probability": 0.4  // "mm-hmm", "I see"
      },

      "conversational_memory": {
        "enabled": true,
        "callback_probability": 0.3     // Reference past topics
      }
    }
  }
}

CLI Commands

Manage your agents with the built-in CLI:

bash 
# Create a new agent from template
npm run agents create my-coach --template coach

# Available templates:
# - coach      (Life coaching, personal growth)
# - analyst    (Research, data analysis)
# - assistant  (Productivity, organization)
# - creative   (Writing, brainstorming)

# List all registered agents
npm run agents list

# Validate agent configuration
npm run agents validate my-coach

# Build agent bundle for deployment
npm run persona:build my-coach
7

Testing Your Agent

Run your agent locally for testing:

bash 
# Start development server with your agent
PERSONA_ID=my-coach npm run dev

# Output:
# Voice agent running on http://localhost:8080
# WebSocket server ready
# Loaded persona: my-coach (Luna)
# Ready to chat!

# Test specific features
npm run test -- --grep "my-coach"

# Validate voice configuration
npm run validate:voices

# Check humanization settings
npm run validate:humanization

Open http://localhost:8080 in your browser and start talking to your agent!

Core Concepts

Cognitive Profiles

Define how your agent reasons and processes information:

  • narrative - Story-driven reasoning (good for coaches)
  • analytical - Data-driven pattern recognition (researchers)
  • empathetic - Emotion-first approach (therapists)
  • pragmatic - Action-oriented solutions (assistants)
  • systematic - Process-oriented structure (organizers)
  • intuitive - Wisdom-based contemplation (philosophers)

Humanization Presets

Choose how natural your agent's speech patterns should be:

  • casual - Friendly, conversational tone
  • professional - Clear, business-appropriate
  • therapeutic - Warm, patient, reflective

Context Builders

Enrich conversations with intelligent context:

  • Memory - Remember user preferences and history
  • Emotion - Detect and respond to emotional state
  • Time - Adapt to time of day and special dates
  • Topics - Track conversation themes
  • Superhuman Insights - Surface patterns users don't see

Advanced Features

🎭 Persona Behaviors

Define greetings, backchannels, self-doubt moments, and vulnerability patterns in JSON files for fine-grained personality control.

📖 Story System

Add personal anecdotes and stories that your agent can naturally weave into conversations when contextually appropriate.

🧠 Domain Knowledge

Provide specialized expertise files that give your agent deep knowledge in specific domains without bloating the main system prompt.

⏰ Time-Based Moods

Configure different energy levels and moods based on time of day or special dates for more authentic interactions.

Next Steps

API Reference

Explore all available endpoints and tools

Example Agents

See real-world persona implementations

Cognitive Profiles Guide

Learn how agents think and reason

Publish to Marketplace

Share your agent with the world