ElevenLabs Launches Conversational AI Bot Builder on Its Platform

ElevenLabs, a prominent startup specializing in AI voice cloning and text-to-speech services, recently unveiled a feature that allows users to build comprehensive conversational AI bots. This new capability, launched on Monday, empowers developers to create sophisticated conversational agents with customizable elements such as voice tone and response length.

Background and Motivation

ElevenLabs has traditionally focused on offering a wide array of voices and AI tools for text-to-speech applications. According to Sam Sklar, the company’s head of growth, many of its clients were already leveraging these tools to develop conversational AI agents. However, they encountered challenges in integrating knowledge bases and managing customer interruptions. To address these issues, ElevenLabs decided to develop a full pipeline for building conversational bots.

Building Conversational Agents

Users can start building a conversational agent by logging into their ElevenLabs account and selecting a template or initiating a new project. Key customization options include:

• Primary Language and First Message: Define the language and initial greeting of the agent.
• System Prompt: Establish the agent’s persona and behavior.
• Large Language Model (LLM): Choose from models like Gemini, GPT, or Claude.
• Response Temperature: Adjust the creativity level of the responses.
• Token Usage Limit: Set limits on the conversation length.

Additional customization features allow users to fine-tune:

• Voice: Select and modify the voice of the agent.
• Latency and Stability: Ensure smooth and reliable interactions.
• Authentication Criteria: Define how the agent verifies user identities.
• Maximum Conversation Length: Set a time limit for interactions.

Knowledge Base and Model Integration

Users can enrich their conversational bots by adding personalized knowledge bases, which can be files, URLs, or text blocks. They also have the flexibility to integrate their own custom LLMs. ElevenLabs provides SDKs compatible with popular programming languages, including Python, JavaScript, React, and Swift. For even more customization, the company offers a WebSocket API.

Data Collection and Evaluation

Developers can specify data points to collect during interactions, such as customer names and emails. They can also define success or failure criteria in natural language to evaluate the performance of the conversations.

Speech-to-Text Capabilities

ElevenLabs is developing speech-to-text capabilities to complement its existing text-to-speech pipeline. Although the speech-to-text API is not yet available as a standalone product, the company plans to consider this in the future. This move could position ElevenLabs as a competitor to established players like Google, Microsoft, Amazon, and specialized APIs such as OpenAI’s Whisper, AssemblyAI, Deepgram, Speechmatics, and Gladia.

Competitive Landscape

ElevenLabs is currently seeking new funding with a valuation target of over $3 billion. In the competitive voice AI market, the company faces rivals like Vapi and Retell, which are also developing conversational agents. Notably, ElevenLabs aims to differentiate itself from OpenAI’s real-time conversational API through its extensive customization options and ability to switch between different models.

Conclusion

With the launch of its conversational AI bot builder, ElevenLabs is expanding its offerings to meet the growing demand for sophisticated AI-driven interactions. By providing robust customization tools and seamless integration capabilities, ElevenLabs positions itself as a leader in the evolving landscape of voice AI technology.