Effect Chat

Your server code uses Effect, and you want a chat that streams tokens. Core chat() gives an async iterable, and its tools return promises. Your Effect services do not reach those tools.

@tanstack/ai-effect connects the two. AI.chat returns an Effect Stream. A tool made with AI.serverTool returns an Effect, and a Layer gives the tool its services.

In this tutorial, the model calls a get_weather tool, and the tool reads a Weather service. This tutorial uses React, TanStack Start, and OpenAI. It takes about 10 minutes.

1. Create a Start app

shell

npx @tanstack/cli@latest create

Pick React. Then install the packages:

shell

npm i @tanstack/ai @tanstack/ai-react @tanstack/ai-openai @tanstack/ai-effect effect

shell

pnpm add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai @tanstack/ai-effect effect

shell

yarn add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai @tanstack/ai-effect effect

shell

bun add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai @tanstack/ai-effect effect

Put your OpenAI key in .env.local:

shell

OPENAI_API_KEY=sk-...

Client and server

The server runs the chat in Effect. It owns the tool and the Layer that gives the tool its service.
The client sends the messages and shows the reply. It does not use Effect.

The next steps build the server first. Then they build the client.

2. Add a service and a tool

Create src/lib/weather.ts:

import { toolDefinition } from '@tanstack/ai'
import * as AI from '@tanstack/ai-effect'
import { Context, Effect, Layer, Schema } from 'effect'

export class Weather extends Context.Service<
  Weather,
  { readonly forecast: (city: string) => Effect.Effect<string> }
>()('Weather') {}

// A fixed forecast. Call a real weather API here.
export const WeatherLive = Layer.succeed(Weather, {
  forecast: (city) => Effect.succeed(`Sunny and 22 C in ${city}`),
})

const getWeatherDef = toolDefinition({
  name: 'get_weather',
  description: 'Get the weather forecast for a city',
  inputSchema: AI.toolSchema(Schema.Struct({ city: Schema.String })),
  outputSchema: AI.toolSchema(Schema.Struct({ forecast: Schema.String })),
})

export const getWeather = AI.serverTool(getWeatherDef, ({ city }) =>
  Effect.gen(function* () {
    const weather = yield* Weather
    return { forecast: yield* weather.forecast(city) }
  }),
)

This file has three parts:

Weather is an Effect service with one function, forecast.
WeatherLive is a Layer that gives a fixed forecast. Put a call to a real weather API here.
getWeather is the tool. AI.serverTool takes a tool definition and a function that returns an Effect.

The type of getWeather keeps Weather as a requirement. If the Effect fails, the model gets a tool error, and the run continues.

3. Add the server route

Create src/routes/api.chat.ts, next to index.tsx. Start maps this file name to the /api/chat path.

import { createFileRoute } from '@tanstack/react-router'
import { chatParamsFromRequest, toServerSentEventsResponse } from '@tanstack/ai'
import * as AI from '@tanstack/ai-effect'
import { openaiText } from '@tanstack/ai-openai'
import { Stream } from 'effect'
import { WeatherLive, getWeather } from '@/lib/weather'

export async function POST({ request }: { request: Request }) {
  const params = await chatParamsFromRequest(request)

  const stream = AI.chat({
    adapter: openaiText('gpt-6.1-sol'),
    messages: params.messages,
    threadId: params.threadId,
    runId: params.runId,
    tools: [getWeather],
  }).pipe(Stream.provide(WeatherLive))

  return toServerSentEventsResponse(Stream.toAsyncIterable(stream))
}

export const Route = createFileRoute('/api/chat')({
  server: {
    handlers: {
      POST,
    },
  },
})

What each piece does:

chatParamsFromRequest reads the messages and the thread id from the POST body.
AI.chat takes the options of core chat(). It returns a Stream of chunks, and Weather is in its requirements.
Stream.provide(WeatherLive) gives the Weather service to the tool. If you remove this line, Stream.toAsyncIterable does not type-check.
toServerSentEventsResponse sends the chunks to the browser as Server-Sent Events.

If the browser closes the request, the stream stops. Then AI.chat aborts the request to OpenAI.

4. Add the chat page

Replace src/routes/index.tsx with this file:

tsx

import { useState } from 'react'
import { createFileRoute } from '@tanstack/react-router'
import { fetchServerSentEvents, useChat } from '@tanstack/ai-react'

function ChatPage() {
  const [input, setInput] = useState('')
  const { messages, sendMessage, isLoading, error, stop } = useChat({
    connection: fetchServerSentEvents('/api/chat'),
    threadId: 'effect-chat',
  })

  const handleSendMessage = () => {
    if (!input.trim()) return
    sendMessage(input.trim())
    setInput('')
  }

  return (
    <div className="mx-auto flex h-screen max-w-2xl flex-col gap-3 p-4 text-white">
      <div className="flex-1 space-y-3 overflow-y-auto">
        {messages.length === 0 ? (
          <p className="text-center text-sm text-gray-400">
            Ask for the weather in a city.
          </p>
        ) : null}
        {messages.map((message) => (
          <div key={message.id}>
            <p className="text-xs text-gray-400">{message.role}</p>
            {message.parts.map((part, index) => {
              if (part.type === 'text') {
                return (
                  <p key={index} className="whitespace-pre-wrap">
                    {part.content}
                  </p>
                )
              }
              if (part.type === 'tool-call') {
                return (
                  <p key={index} className="text-sm text-orange-400">
                    Called {part.name}
                  </p>
                )
              }
              return null
            })}
          </div>
        ))}
      </div>
      {error ? <p className="text-sm text-red-400">{error.message}</p> : null}
      <div className="flex gap-2">
        <textarea
          value={input}
          onChange={(event) => setInput(event.target.value)}
          onKeyDown={(event) => {
            if (event.key === 'Enter' && !event.shiftKey) {
              event.preventDefault()
              handleSendMessage()
            }
          }}
          placeholder="What is the weather in Oslo?"
          rows={1}
          disabled={isLoading}
          className="flex-1 resize-none rounded-lg border border-gray-700 bg-gray-800 px-3 py-2 text-sm"
        />
        {isLoading ? (
          <button
            type="button"
            onClick={stop}
            className="rounded-lg bg-red-600 px-4 py-2 text-sm"
          >
            Stop
          </button>
        ) : (
          <button
            type="button"
            onClick={handleSendMessage}
            disabled={!input.trim()}
            className="rounded-lg bg-orange-500 px-4 py-2 text-sm disabled:opacity-50"
          >
            Send
          </button>
        )}
      </div>
    </div>
  )
}

export const Route = createFileRoute('/')({
  component: ChatPage,
})

The client does not use Effect:

useChat sends the messages to /api/chat and updates messages as tokens arrive.
A tool-call part shows each time the model calls get_weather.

Run the app and send "What is the weather in Oslo?". The page shows "Called get_weather". Then the reply streams in with the forecast from your Weather service.

effect-chat

You have a streaming chat with an Effect server, and a tool that gets its service from a Layer.

Next, pick one: