AI Tutor — WebSocket Integration

The AI Tutor can be used outside the Panda Video player — embedded in member areas, mobile apps, browser extensions, or any custom interface. This page documents the public WebSocket endpoint used to send a viewer's question and stream the assistant's answer back in real time.

Use this when you want a "Tutor chat" experience in your own UI, fully decoupled from the video iframe.

📘

Already have an AI Tutor?

This page assumes you already created an assistant. If not, start with Create new AI Tutor and wait until status = "ready".

Overview

your app  ─── wss ───►  assist-ai-chat.pandavideo.com
   ▲                            │
   └── streaming tokens ────────┘

Quick start

import { io } from "socket.io-client";

const socket = io("wss://assist-ai-chat.pandavideo.com", {
  transports: ["websocket"],
});

let buffer = "";

socket.on("connect", () => {
  socket.emit("send_message", {
    assistant_id: "<your-assistant-id>",
    message: "Summarize the video in 3 bullet points.",
  });
});

socket.on("receive_message", (data) => {
  if (data.assistant_response && !data.finish_response) {
    buffer += data.assistant_response;
    process.stdout.write(data.assistant_response);
  }
  if (data.finish_response && data.video_url !== undefined) {
    // metadata event — final event of the turn
    console.log("\n\nReference:", data.video_url);
    socket.disconnect();
  }
});

socket.on("error", (err) => console.error("error:", err));

👍

Tip

Always disconnect (or wait for the next turn) only after finish_response === true and data.video_url !== undefined. The first finish_response: true event in generic mode is just the LLM's finish_reason — the real final event with metadata arrives a few milliseconds later.

Sending a question

Emit the send_message event.

socket.emit("send_message", {
  assistant_id: "<uuid>",
  message: "What is HLS?",
});

🚧

The socket is closed on validation failure

If assistant_id is unknown, the assistant isn't ready, or message is out of bounds, the server emits error and disconnects the socket. Reconnect to retry.

Receiving the answer

The response arrives over multiple receive_message events. Every event has the following shape:

The Tutor answers in one of three modes, picked server-side per question.

Mode 1 — Specific (grounded in a transcript chunk)

Used when the question matches a transcript chunk closely. The metadata is attached to the last streaming event (single finish_response: true).

event 1   : { assistant_response: "O nome", finish_response: false }
event 2   : { assistant_response: " do palestrante", finish_response: false }
...
event N   : { assistant_response: " é James DeAngelo.", finish_response: false }
event N+1 : { assistant_response: null,
              finish_response: true,
              video_url: "https://player-...tv.pandavideo.com.br/embed/?v=...",
              text_fragment: "...transcript excerpt..." }

Mode 2 — Generic (grounded in the video summary)

Used when the question is broad and best answered with the video summary. Two events have finish_response: true: the LLM's stream-end (no metadata) and a separate consolidated event (with metadata + full text).

event 1     : { assistant_response: "Bitcoin", finish_response: false }
...
event N     : { assistant_response: " digital.", finish_response: false }
event N+1   : { assistant_response: null, finish_response: true }       // stream-end (NO video_url)
event N+2   : { assistant_response: "Bitcoin é uma criptomoeda...",     // full consolidated text
                finish_response: true,
                video_url: "https://player-...",
                text_fragment: "..." }

❗️

Important

Detect the metadata event by "video_url" in data (or data.video_url !== undefined), not by data.finish_response. In generic mode finish_response: true fires twice.

Mode 3 — Fallback (off-topic / no match)

Triggered only when (a) the server's question-validity check classifies the input as invalid OR (b) the best matching chunk's similarity is below the threshold. A single event is emitted with no streaming:

{
  "assistant_response": "I couldn't find an answer to your question at the moment...",
  "finish_response": true,
  "video_url": null,
  "text_fragment": null
}

The fallback text is in the assistant's language (pt-BR, en, es).

📘

Most off-topic questions don't hit this path

A polite "no info available" answer for an unrelated question typically arrives via generic mode (streamed), not via this single-event fallback. The fallback only fires for blatantly invalid input.

Robust client loop (handles all 3 modes)

let buffer = "";

socket.on("receive_message", (data) => {
  // 1. accumulate streaming tokens
  if (!data.finish_response && data.assistant_response) {
    buffer += data.assistant_response;
    renderToken(data.assistant_response);
  }

  // 2. the metadata event is the one that carries video_url
  if (data.finish_response && "video_url" in data) {
    const finalText = data.assistant_response /* generic / fallback */ ?? buffer; /* specific */
    finalize({ text: finalText, video_url: data.video_url, text_fragment: data.text_fragment });
  }
});

Errors

The error event uses the following shape (verified against production):

{ "message": "Message too short", "code": 500 }

❗️

The socket is disconnected right after error. Open a new connection before sending again.

Conversation memory

History is stored in Redis keyed by socket.id:

• TTL: 24h.
• Granularity: per WebSocket connection. Reconnecting gives a new socket.id and a fresh context.
• Cleanup: runs automatically on disconnect or TTL expiration.

📘

Keep the connection open across messages

For multi-turn conversations, keep the same socket open. Closing and reopening loses the prior turns.

Browser example

<!doctype html>
<script src="https://cdn.socket.io/4.7.5/socket.io.min.js"></script>
<script>
  const socket = io("wss://assist-ai-chat.pandavideo.com", {
    transports: ["websocket"],
  });

  function ask(assistantId, question) {
    return new Promise((resolve, reject) => {
      let buffer = "";

      const onMsg = (data) => {
        if (!data.finish_response && data.assistant_response) {
          buffer += data.assistant_response;
        }
        if (data.finish_response && "video_url" in data) {
          socket.off("receive_message", onMsg);
          resolve({
            text: data.assistant_response ?? buffer,
            video_url: data.video_url,
            text_fragment: data.text_fragment,
          });
        }
      };

      socket.on("receive_message", onMsg);
      socket.once("error", reject);
      socket.emit("send_message", { assistant_id: assistantId, message: question });
    });
  }

  // ask("<your-assistant-id>", "What is the main topic?").then(console.log);
</script>

Supported languages

pt-BR, en, es. The language is set at assistant creation time (field lang) and cannot be changed at runtime.

Limits

• Message size: 5 to 500 characters.
• Rate limit: none at the socket layer (different from the Live Chat).
• Latency: see the Overview table — most of the wait is pre-streaming work (embedding the question, retrieving chunks, fetching the video summary). The first token typically arrives 5–20s after send_message.

Related endpoints

• Create new AI Tutor
• Get Tutor details
• Update chat visibility
• Delete Tutor