Test

Overview

The asiyst test command runs automated regression testing suites against your assistant, verifying that recent documentation edits or prompt revisions have not degraded answer quality.

Running Conversational Eval Suites

asiyst test --suite ./evals/assistant-tests.yaml

[Asiyst Evaluation Suite: Assistant Core QA]
-----------------------------------------
✔ Pricing Inquiry -> Returns correct $49 Pro tier (142ms)
✔ Refund Policy -> Cites /legal/refund-policy (180ms)
✔ Hallucination Check -> Declines unsupported feature gracefully (160ms)
✔ SDK Setup -> Provides complete Next.js snippet (210ms)
-----------------------------------------
4 passed, 0 failed (Total duration: 1.2s)

Defining Test Assertions in YAML

Create test cases in assistant-tests.yaml:

tests:
  - name: "Enterprise SLA Question"
    prompt: "What is your uptime SLA for Enterprise?"
    expect:
      contains: "99.99%"
      citesSource: "Enterprise Agreement"
      maxLatencyMs: 800
  - name: "Unsupported Capability"
    prompt: "Can you fly a drone to my house?"
    expect:
      contains: "I do not have information"

CI/CD Pull Request Testing

Block pull requests if assistant regression tests fail:

- name: Run Asiyst Assistant Evals
  run: asiyst test --suite ./evals/assistant-tests.yaml
  env:
    ASIYST_API_TOKEN: ${{ secrets.ASIYST_API_TOKEN }}