Collecting user feedback

User feedback is in beta

User feedback is currently in beta. We'd love to hear your feedback as we develop this feature:

To collect user feedback for LLM traces, PostHog provides a native integration with Surveys. The Surveys integration enables you to collect user feedback and thumbs up/down responses attached directly to a given trace.

How it works

  1. Start by creating a survey for a group of LLM traces.
  2. In your app, collect survey responses with an $ai_trace_id property attached to the events.
  3. View user feedback per-trace within AI Observability, or view all feedback within Surveys.

For the common thumbs up/down use-case, we provide a React hook useThumbSurvey to quickly attach a thumb survey and optional followup questions to your existing UI. In other cases, you can manually send survey-related events to attach them to your trace.

Creating a survey

On any top-level trace (or pseudo-trace) event, navigate to the Feedback tab to create a survey.

LLM trace Feedback tab

The survey will begin with a thumbs up/down question. Toggle on Ask follow-up on thumbs down to ask a follow-up question when the user responds with "thumbs down".

LLM trace survey creation

For more customization you can edit this survey post-creation, or create a new survey. However, complex surveys may not work with our useThumbSurvey hook and require manual event capture.

Click Create & continue to create your survey and view implementation instructions.

Integrating with your app

In React apps, use the useThumbSurvey hook. For any other app, or for more control, capture the feedback events yourself.

React: use the useThumbSurvey hook

To quickly integrate with your React app, we provide a hook useThumbSurvey that exposes a method to submit survey responses, the user's response, and a ref to position followup question pop-ups.

Quickstart

Note: This hook requires @posthog/react 1.7.1+, which is bundled with posthog-js 1.345.1+

Initialize the hook with the following options:

typescript
export interface UseThumbSurveyOptions {
/** ID of the target PostHog survey */
surveyId: string;
/** Configure the position of the pop-up for followup questions, if applicable. Defaults to SurveyPosition.NextToTrigger */
displayPosition?: SurveyPosition;
/** Custom event properties to pass with each survey result */
properties?: Record<string, any>;
/** Callback on thumb button click */
onResponse?: (response: "up" | "down") => void;
/** Disable automatically emitting `survey shown` on hook mount. Defaults to false. */
disableAutoShownTracking?: boolean;
}

It returns the following interface:

typescript
export interface UseThumbSurveyResult {
/** Call this to submit a survey response, with value 'up' or 'down' */
respond: (value: "up" | "down") => void;
/** User's response value, available after submission. Not persisted across mounts. */
response: "up" | "down" | null;
/** Ref to attach to the trigger element for positioning the followup survey popup */
triggerRef: RefCallback<HTMLElement>;
/** Method to manually trigger a `survey shown` event. Only available when disableAutoShownTracking is true. */
trackShown?: () => void;
}

A simple usage of this hook, including a followup question pop-up, looks like this:

typescript
// requires @posthog/react 1.7.1+ (bundled with posthog-js 1.345.1+)
import { useThumbSurvey } from '@posthog/react/surveys'
function HedgehogBotResponse({ traceId }: { traceId: string }) {
const { respond, response, triggerRef } = useThumbSurvey({
surveyId: '019bb5a3-1677-0000-63dd-00f241c1710a', // ID for the survey you just created
properties: {
$ai_trace_id: traceId, // your generated trace ID
// add any other custom properties here to be attached to `survey sent` and `survey shown` events
},
})
return (
<div>
<ChatBubble>You're absolutely right! I should have been using PostHog all along.</ChatBubble>
{/* PostHog followup pop-up anchors to triggerRef. If not provided, defaults to showing on the center of the page. */}
<div ref={triggerRef}>
<p>Was this response helpful?</p>
<button onClick={() => respond('up')} >
<ThumbsUpIcon className={response === 'up' ? 'active' : ''} />
</button>
<button onClick={() => respond('down')} >
<ThumbsDownIcon className={response === 'down' ? 'active' : ''} />
</button>
</div>
</div>
)
}

Impression tracking

By default, the hook emits a survey shown (impression) event on mount.

Duplicate impression events will not impact your ability to see responses within AI Observability, but may skew metrics viewed directly from Surveys.

To disable this behavior, initialize the hook with disableAutoShownTracking: true and manually call trackShown() when you consider your survey to be "shown."

Response persistence

The hook's response value is not persisted across mounts. We plan to improve this soon, but in the meantime, you'll need to set up your own persistent response tracking if needed.

Question indices

The hook submits survey responses using the legacy index-based tracking for survey questions. If you add or re-order survey questions from the Surveys UI, you may experience unexpected behavior. See Implementing custom surveys for more information.

Survey compatibility

To use this hook, your survey must begin with a thumbs up/down question. All other surveys features are supported, such as followup questions and question branching.

For more information, see the hook source here on GitHub, or a real implementation in the PostHog codebase here.

Capture feedback events manually

If you are not building a React app or otherwise need more control (such as a completely custom followup UI), you can capture user feedback events manually.

In summary, you'll emit these events with $ai_trace_id properties:

  1. Emit a survey shown event when a user sees your survey
  2. Emit a survey sent event as a user answers questions, making sure the final event carries every answer collected so far

Step 1: Track impressions

While not required, we recommend emitting survey shown events when a user sees your survey. This enables two things:

  1. Accuracy when viewing all metrics in Surveys
  2. Heads-up messages within LLM traces that indicate a user saw a survey, but did not respond
LLM trace survey impression with no response

survey shown events must have the following properties:

PropertyDescription
$survey_idYour survey ID
$ai_trace_idYour generated trace ID

Note: we do not yet support an easy way to check whether a user has responded to a given survey/trace combination. Because of this, without your own persistent tracking, you may end up with duplicate survey shown events.

Duplicate events will not impact your ability to view responses per trace, but will skew impression & response rate metrics on your Survey overview.

Step 2: Collect responses

For a multi-question survey, send one survey sent event per question as answers come in, and give every event the same $survey_submission_id. Mark the last event with $survey_completed: true.

Important: PostHog displays the completed event as the response for a submission, so that final event must include every answer collected so far, not just the current question's. If an earlier question's $survey_response_{questionId} property is missing from the completed event, that answer won't show up in the response or the question breakdown. The example below shows the pattern.

survey sent events may have the following properties (* = required):

PropertyDescription
$survey_id *Your survey ID
$survey_response_{questionId} *A question's response value, where {questionId} is that question's ID. Include one property per answered question. On the completed event, include every response collected in the submission.
$ai_trace_id *ID for the trace corresponding to this survey
$survey_submission_idA unique ID you generate to link a submission's events together. Reuse the same value across every event in the submission.
$survey_completedBoolean for whether this is the final event in the submission. The completed event is the one PostHog displays, so it must carry every answer.

To get survey question IDs, you have two options:

  1. Go to Surveys, find your survey, and check the Overview tab. This will show the full survey object, including each question's ID.
  2. Call the PostHog SDK's getSurveys method to retrieve your survey, then extract the question IDs from the response.

To learn more about the properties available for survey events, or how to use the getSurveys method, see Implementing custom surveys.

Example

An example sequence for a thumbs up/down survey with a followup may look like this:

typescript
const traceId = 'your-generated-trace-id'
const surveyId = 'your-survey-id'
// (Optional) Track when the survey is shown to the user
posthog.capture('survey shown', {
$survey_id: surveyId,
$ai_trace_id: traceId,
})
// Generate a unique ID to link `survey sent` events into a single user feedback event
const submissionId = crypto.randomUUID()
// User clicked "thumbs down"
posthog.capture('survey sent', {
$survey_id: surveyId,
[`$survey_response_${firstQuestionId}`]: 2, // 1 = thumbs up, 2 = thumbs down
$ai_trace_id: traceId,
$survey_submission_id: submissionId,
$survey_completed: false,
})
// User submitted follow-up text after the "thumbs down" response.
// This is the completed event, so it re-sends the thumbs response
// alongside the follow-up text, so the thumbs rating still shows.
posthog.capture('survey sent', {
$survey_id: surveyId,
[`$survey_response_${firstQuestionId}`]: 2, // re-send the thumbs response
[`$survey_response_${secondQuestionId}`]: 'the AI hallucinated hedgehogs everywhere',
$ai_trace_id: traceId,
$survey_submission_id: submissionId, // must match the previous event's $survey_submission_id
$survey_completed: true,
})

Viewing survey responses

You can view survey responses per-trace within AI Observability, or per-survey within Surveys.

View all feedback in Surveys

Navigate to Surveys and open your LLM feedback survey. You'll see aggregated survey metrics at the top, such as overall thumbs up/down ratings:

Survey thumbs up down ratings

You can also view all responses in a table, and quickly link to the related trace by clicking the AI Observability icon:

Survey thumbs up down ratings

View all traces with feedback

Use this SQL expression to filter your LLM traces down to only those with attached survey responses:

SQL
properties.$ai_trace_id IN (
SELECT properties.$ai_trace_id
FROM events
WHERE event = 'survey sent'
)
SQL expression for filtering LLM traces with feedback

View individual trace feedback

For an individual trace, navigate to the Feedback tab to see survey responses.

If there are no survey sent or survey shown events associated with the given trace, you'll see the option to create a new survey, or select an existing one to see implementation instructions.

Otherwise you'll see the survey response, or a message indicating a survey was shown but no response was sent.

Negative feedback on trace event with followup

Best practices

Organizing feedback

We recommend creating one survey per logical "group" of traces.

For example, consider a product that offers users two AI generations:

  1. Chatbot to help query the docs
  2. Agent to help build custom dashboards

In this case, we'd recommend creating two separate surveys: one for the docs chatbot, and another for the dashboard agent.

This would enable you to quickly view aggregate feedback for all docs chat traces or custom dashboard generation traces via the Surveys UI.

If you use a single survey for all traces, it will be difficult to parse them per "group" or "user action" without adding custom event properties and building your own dashboards.

Still have questions?

Was this page useful?