Evaluation reports

Evaluation reports summarize your evaluation results over time and send them to email or Slack, so you can track quality without reading every result. PostHog creates a default report configuration for every evaluation you make.

Reports are available for all evaluation types – LLM-as-a-judge, code-based, and sentiment analysis – on any target.

Reports also power Self-driving. Turn on the AI observability source in your inbox settings, and report findings arrive as signals.

Default report configuration

The auto-created report uses these defaults:

SettingDefault valueDescription
FrequencyCount-basedGenerates a report after a set number of evaluation runs
Trigger threshold100A report is generated every 100 new evaluation results
Cooldown1 hourMinimum hours between count-triggered reports (1 – 24 hours)
Daily run cap10Maximum report runs per day
EnabledYesReports start generating immediately
Delivery targetsNoneReports are viewable in the UI but no notifications are sent

Reports are viewable directly in the evaluation's Reports tab without configuring any delivery targets.

Report metrics by evaluation type

Reports display different metrics depending on your evaluation's output type:

  • Boolean evaluations (LLM-as-a-judge and code-based) – Pass rate, pass/fail/N/A counts, total runs, and comparison to the previous reporting period
  • Numeric score and categorical evaluations – Pass rate and pass/fail/N/A counts, based on the evaluation's passing rule. Reports need a passing rule.
  • Sentiment analysis evaluations – Positive, neutral, and negative counts with distribution percentages, total runs, and comparison to the previous reporting period

Customizing reports

You can adjust the report configuration after creating an evaluation:

  • Change the trigger threshold to generate reports more or less frequently
  • Adjust the minimum hours between reports (1 – 24) to control how often count-triggered reports can fire
  • Add email or Slack delivery targets to receive notifications when reports are generated
  • Disable the report if you don't need automated summaries

Still have questions?

Was this page useful?