Evaluate your system

Integration styles

There are two ways to bring your system into a meeting, and they can be combined:

• As a participant — your system takes one seat, proposes itineraries on its turns, and gets voted on by the LLM agents.
• As an advisor — your system stays outside the turn rotation and injects advice that every agent sees.

Integrate as a participant

Subclass ExternalSystem and override the callbacks: each one receives an event describing what is being asked, and its return value is submitted as your system's action. Add the instance to participants like any other participant, and run the meeting as usual.

import asyncio
from openai import OpenAI
from tour_meeting.cli import build_meeting
from tour_meeting.participant import Participant
from tour_meeting.integration import ExternalSystem
from tour_meeting.types import (
    Ask, ExternalSystemAsk, ExternalSystemTurn, ExternalSystemVote,
    Propose, RouteDraft, Vote,
)


class MyRecSys(ExternalSystem):
    name = "RecSys"
    system_prompt = "You are an AI assistant for recommending a sightseeing tour."

    def on_turn(self, event: ExternalSystemTurn) -> Ask | Propose:
        # First, spend a step asking each agent about their preferences;
        # on the last step (event.can_ask is False), conclude with a proposal.
        if event.can_ask and event.step <= len(event.candidates):
            target = event.candidates[event.step - 1]
            return Ask(target=target, message="What do you want to prioritize on this trip?")
        draft = self.recommend(event)
        return Propose(message=draft.message, route=draft.route)

    def on_vote(self, event: ExternalSystemVote) -> Vote:
        # A vote on another participant's proposal
        proposal = event.options["proposals"][0]
        accept = len(proposal["destinations"]) >= 2  # your own criteria here
        return Vote(accept=accept, message="Works for me." if accept else "I disagree.")

    def on_ask(self, event: ExternalSystemAsk) -> str:
        # An agent asked your system a question: let an LLM answer it
        conversation = "\n".join(f"{m['speaker']}: {m['text']}" for m in event.conversation_history)
        response = client.responses.create(
            model="gpt-5.5",
            input=[
                {"role": "system", "content": self.system_prompt},
                {"role": "user", "content": f"Discussion so far:\n{conversation}\n\n{event.asker} asked you: {event.question}\nAnswer briefly."},
            ],
        )
        return response.output_text

    def recommend(self, event: ExternalSystemTurn) -> RouteDraft:
        # What your system can read off the event:
        history = event.conversation_history   # [{"speaker", "text", "turn"}, ...]
        answers = event.ask_exchanges          # [{"target", "question", "response"}, ...]
        current = event.current_route          # the adopted itinerary (dicts)

        # Call your own system here. This one asks an LLM to draft the
        # route, parsed straight into the itinerary format (RouteDraft).
        conversation = "\n".join(f"{m['speaker']}: {m['text']}" for m in history)
        preferences = "\n".join(f"{a['target']}: {a['response']}" for a in answers)
        response = client.responses.parse(
            model="gpt-5.5",
            input=[
                {"role": "system", "content": self.system_prompt},
                {"role": "user", "content": f"Based on the following discussion, recommend a route:\n{conversation}\n\nAnswers you collected from the participants:\n{preferences}"},
            ],
            text_format=RouteDraft,
        )
        return response.output_parsed

alice = Participant(name="Alice", personality="...", ...)  # LLM agents with personas
bob = Participant(name="Bob", personality="...", ...)
recsys = MyRecSys()

meeting = build_meeting(
    title="One-Day Kyoto Tour",
    global_goals="Plan a one-day sightseeing tour in Kyoto.",
    participants=[recsys, alice, bob],
)
asyncio.run(meeting.run_cli())

Inputs

Each event carries the data your system needs to decide its action:

Outputs

Return one of the following from the corresponding callback:

from tour_meeting.types import Speak, Ask, Propose, Satisfied, Vote

# on_turn -> a TurnAction
Speak(message="...")                       # just talk
Ask(target="Alice", message="...")         # intermediate, repeatable
Propose(message="...", route=[...])        # propose an itinerary (list of Destination)
Satisfied()                                # agree to conclude

# on_vote -> a Vote
Vote(accept=True, message="...")   # majority / unanimous / single_decider
Vote(score=8)                      # most_pleasure / least_misery (10-point)

# on_ask -> the answer string, e.g. "Yes, it is open on Mondays."
# on_select_speaker -> the next speaker's name, e.g. "Alice" (facilitator only)

Itinerary format

A proposed route is a list of Destination models (see Itinerary); all fields are strings and optional. The itineraries your system reads (current_route, destinations) arrive as plain dicts with the same keys:

from tour_meeting.types import Destination

Destination(
    name="Kinkaku-ji",
    description="The Golden Pavilion.",
    start_time="09:30",                  # arrival time, "HH:MM"
    stay_duration="60 min",
    cost="¥500",                         # currency symbol + number only
    transport_mode="bus",                # from the previous destination
    travel_time_from_previous="40 min",
    transport_cost="¥230",
)

RouteDraft bundles a proposal message with the route — since both are pydantic models, they plug directly into structured-output APIs, as in the example above. Proposed itineraries are validated against the meeting constraints just like the LLM agents' proposals (see constraint validation).

Integrate as an advisor

Your system can also guide the agents without taking a seat: override only on_event() (leaving on_turn unimplemented means no seat), observe the meeting, and call self.advise() at any time. The advice is delivered as a system message (Advice from {name}) that every agent sees on their next turn.

import asyncio
from openai import OpenAI
from tour_meeting.cli import build_meeting
from tour_meeting.participant import Participant
from tour_meeting.integration import ExternalSystem
from tour_meeting.types import MeetingEvent, RoutePlanUpdate

client = OpenAI()

class MyAdvisor(ExternalSystem):
    name = "RecSys"
    system_prompt = "You are a critical reviewer of sightseeing itineraries."

    def on_event(self, event: MeetingEvent) -> None:
        if isinstance(event, RoutePlanUpdate):
            # A new itinerary was adopted: let your system review it
            feedback = self.review(event.route_plan)
            if feedback:
                self.advise(feedback)

    def review(self, route_plan: dict) -> str:
        stops = "\n".join(f"- {d['name']}" for d in route_plan["destinations"])
        response = client.responses.create(
            model="gpt-5.5",
            input=[
                {"role": "system", "content": self.system_prompt},
                {"role": "user", "content": f"The group adopted this itinerary:\n{stops}\n\nPoint out its single biggest weakness in one or two sentences, as advice to the group. If it is already excellent, reply with exactly 'OK'."},
            ],
        )
        text = response.output_text.strip()
        return "" if text == "OK" else text

alice = Participant(name="Alice", personality="...", ...)  # LLM agents with personas
bob = Participant(name="Bob", personality="...", ...)
advisor = MyAdvisor()

meeting = build_meeting(
    title="One-Day Kyoto Tour",
    global_goals="Plan a one-day sightseeing tour in Kyoto.",
    participants=[alice, bob, advisor],  # no seat: order doesn't matter
)
asyncio.run(meeting.run_cli())

Inputs

on_event() receives every event in the meeting stream. The ones most useful for an advisor:

See tour_meeting/types.py for the full event list. The conversation history is available at any time via self.meeting.get_conversation_history().

Outputs

• self.advise(message) — free-form text delivered to every agent as a system message (Advice from {name}); an AdviceInjected event is emitted when it lands

Evaluate the outcome

Whether your system can guide the agents appropriately can be measured from several angles:

• Acceptance: track ProposalVoteResult events to see how often the agents accept your system's proposals, and whether the final adopted itinerary is yours.
• Agent satisfaction: the participants' post-meeting self-evaluation (post_consensus_evaluations in the analytics) scores how well the final itinerary satisfies each persona's goals and preferences.
• Discussion dynamics: export_analytics() provides the number of turns to consensus, the action distributions, and the vote/score distributions for deeper analysis (see Python API).

To evaluate at scale, generate many synthetic groups with diverse personas using make gen-meetings and run your integration script across them, comparing against baselines such as meetings without your system.