← All features

Run the agentsAvailable in v0.3.0

Scores from Epoch AI, or your own Artificial Analysis key

New in v0.3.0: scores come from Epoch AI, with no key and no account. With your own Artificial Analysis key, oah uses those scores instead. Each release ships an Epoch AI snapshot, a catalog rebuild refreshes it over HTTPS, and the request sends nothing about your project, your tasks or you.

When you need it

Model choice by score is only as good as its data, and fetching that data must not leak anything about your work.

How it works

  1. At a catalog rebuild — after a CLI update, after [models] ttl_hours (24 by default), or with oah models --refresh — oah fetches https://epoch.ai/data/benchmarks.csv when its stored copy is older, missing or damaged, with a 10 second timeout and a size limit.
  2. With ARTIFICIAL_ANALYSIS_API_KEY stored, oah also asks Artificial Analysis, sending the key only in the x-api-key header, and keeps one index per matched model.
  3. Each engine uses one source per launch: Artificial Analysis when its copy scores the newest listed model of every known family, else the latest Epoch AI fetch, else the shipped snapshot. Sources never mix.
  4. Copies stay in oah's state folder (models/benchmarks-epoch.json, benchmarks-aa.json); a damaged or changed file counts as absent (OAH_SCORES_FILE_INVALID).
  5. [models] fetch_scores = false stops every fetch, and the shipped snapshot chooses.

Example

Store your own Artificial Analysis key in the macOS keychain (from Keys).

security add-generic-password -s oah -a ARTIFICIAL_ANALYSIS_API_KEY -w

Requirements

  • Requires v0.3.0 or later.
  • No key for Epoch AI. Optional: your own ARTIFICIAL_ANALYSIS_API_KEY in the environment or the OS keychain.

Limits and recovery

  • Unlike the Jev re-ranker, the Artificial Analysis key alone turns this on, because [models] fetch_scores is true by default.
  • Matching Artificial Analysis rows by name follows its documentation and is not yet confirmed against a live response.
  • A failed fetch keeps the last good copy and says so once (OAH_SCORES_REFRESH_FAILED); a launch never fails because of it.
  • Remove the key and oah stops using the data and deletes its copy at the next refresh.

Original documents

Related