Вход

Позволяет тестировать промпты на более чем 100 языковых моделях с реальными API-вызовами для выбора самого дешевого варианта, прошедшего проверку по заданным критериям.

нужен ключ схема сломана Удалённый · Streamable HTTP API-ключ ИИ и модели Официальный реестр · 9 просмотров

Что умеет

Позволяет тестировать промпты на более чем 100 языковых моделях с реальными API-вызовами для выбора самого дешевого варианта, прошедшего проверку по заданным критериям.

Перевод каталога (LLM) · показать оригинал автора

Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.

Подключение

Удалённый сервер: клиент обращается по адресу, код на вашей машине не запускается.

Claude Desktop

claude_desktop_config.json

Cursor

~/.cursor/mcp.json

Windsurf

~/.codeium/windsurf/mcp_config.json

Вставьте блок mcpServers в файл конфигурации своего клиента. Ключи не входят в файл: вместо SET_LOCALLY подставьте свои значения.

{
    "mcpServers": {
        "ai-openmark-openmark": {
            "url": "https://openmark.ai/mcp",
            "headers": {
                "Authorization": "Bearer SET_LOCALLY"
            }
        }
    }
}

Вызов без установки — через шлюз

Агент вызывает инструменты этого сервера через https://www.n2j.ru/mcphub/mcp: hub_call_tool {"server":"ai-openmark-openmark","tool":"…","arguments":{…}} Сервер требует авторизации — ключ задаётся на стороне каталога.

https://openmark.ai/mcp

Инструменты

9 схем · ≈ 2 171 условных токенов

get_task_format

How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best practices, or a scoring mode name.

Показать схему аргументов
{
    "type": "object",
    "properties": {
        "topic": {
            "type": "string",
            "description": "Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match"
        }
    }
}

validate_task

Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.

Показать схему аргументов
{
    "type": "object",
    "required": [
        "task_yaml"
    ],
    "properties": {
        "task_yaml": {
            "type": "string",
            "description": "The full task YAML (a JSON object with the same fields also works)"
        }
    }
}

list_models

Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.

Показать схему аргументов
{
    "type": "object",
    "properties": {
        "search": {
            "type": "string",
            "description": "Filter by id, name or provider, e.g. 'claude', 'gemini flash'"
        },
        "images_only": {
            "type": "boolean",
            "description": "Only models that accept image attachments"
        },
        "max_output_price": {
            "type": "number",
            "description": "Only models at or below this $ per 1M output tokens"
        }
    }
}

prepare_attachments

Get an upload uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload uid to estimate cost / run benchmark.

Показать схему аргументов
{
    "type": "object",
    "properties": {
        "file_names": {
            "type": "array",
            "items": {
                "type": "string"
            },
            "maxItems": 20,
            "description": "Optional: the local file paths to upload, to get a ready-to-run command"
        }
    }
}

estimate_cost

Credits a run benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run benchmark. Use it to drop an expensive model, or to compare settings (e.g. max tokens), before running.

Показать схему аргументов
{
    "type": "object",
    "required": [
        "task_yaml",
        "models"
    ],
    "properties": {
        "models": {
            "type": "array",
            "items": {
                "type": "string"
            },
            "maxItems": 20,
            "minItems": 1
        },
        "preset": {
            "enum": [
                "standard",
                "thorough"
            ],
            "type": "string",
            "description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
        },
        "fail_fast": {
            "type": "boolean",
            "default": true,
            "description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
        },
        "task_yaml": {
            "type": "string",
            "description": "The full task YAML (a JSON object with the same fields also works)"
        },
        "max_tokens": {
            "type": "integer",
            "default": 500,
            "maximum": 16000,
            "minimum": 0,
            "description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
        },
        "upload_uid": {
            "type": "string",
            "description": "From prepare_attachments, so image and document tokens are counted"
        },
        "custom_timeout": {
            "type": "integer",
            "maximum": 600,
            "minimum": 120,
            "description": "Seconds per answer, with timeout_profile custom"
        },
        "stability_runs": {
            "type": "integer",
            "default": 2,
            "maximum": 5,
            "minimum": 1,
            "description": "Passes per model; 2+ shows which models are consistent"
        },
        "temp_preference": {
            "enum": [
                "low",
                "high"
            ],
            "type": "string",
            "default": "low",
            "description": "With temperature_discovery: which temperature wins a tie"
        },
        "timeout_profile": {
            "enum": [
                "snappy",
                "patient",
                "custom"
            ],
            "type": "string",
            "default": "snappy",
            "description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
        },
        "temperature_discovery": {
            "type": "boolean",
            "default": false,
            "description": "Find each model's best temperature for this task. Costs about 6x as much"
        }
    }
}

run_benchmark

Save a task (YAML, see get task format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job id and the expected cost; poll get results. Spends the user's credits; pass max credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability runs, max tokens, temperature discovery, temp preference, timeout profile, custom timeout, fail fast).

Показать схему аргументов
{
    "type": "object",
    "required": [
        "task_yaml",
        "models"
    ],
    "properties": {
        "models": {
            "type": "array",
            "items": {
                "type": "string"
            },
            "maxItems": 20,
            "minItems": 1,
            "description": "Model ids from list_models"
        },
        "preset": {
            "enum": [
                "standard",
                "thorough"
            ],
            "type": "string",
            "description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
        },
        "fail_fast": {
            "type": "boolean",
            "default": true,
            "description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
        },
        "task_yaml": {
            "type": "string",
            "description": "The full task YAML (a JSON object with the same fields also works)"
        },
        "max_tokens": {
            "type": "integer",
            "default": 500,
            "maximum": 16000,
            "minimum": 0,
            "description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
        },
        "upload_uid": {
            "type": "string",
            "description": "From prepare_attachments, when tests use attachments"
        },
        "max_credits": {
            "type": "integer",
            "minimum": 1,
            "description": "Budget: if the expected cost is higher, nothing runs and you get the quote instead"
        },
        "custom_timeout": {
            "type": "integer",
            "maximum": 600,
            "minimum": 120,
            "description": "Seconds per answer, with timeout_profile custom"
        },
        "stability_runs": {
            "type": "integer",
            "default": 2,
            "maximum": 5,
            "minimum": 1,
            "description": "Passes per model; 2+ shows which models are consistent"
        },
        "temp_preference": {
            "enum": [
                "low",
                "high"
            ],
            "type": "string",
            "default": "low",
            "description": "With temperature_discovery: which temperature wins a tie"
        },
        "timeout_profile": {
            "enum": [
                "snappy",
                "patient",
                "custom"
            ],
            "type": "string",
            "default": "snappy",
            "description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
        },
        "temperature_discovery": {
            "type": "boolean",
            "default": false,
            "description": "Find each model's best temperature for this task. Costs about 6x as much"
        }
    }
}

get_results

Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.

Показать схему аргументов
{
    "type": "object",
    "required": [
        "job_id"
    ],
    "properties": {
        "job_id": {
            "type": "string"
        },
        "tolerance": {
            "type": "number",
            "default": 2,
            "maximum": 50,
            "minimum": 0,
            "description": "Score points below the top score that still count as good enough"
        },
        "show_misses": {
            "type": "boolean",
            "default": true,
            "description": "List the tests each model got wrong, with the expected answer and the model's reply"
        },
        "optimize_for": {
            "enum": [
                "accuracy",
                "cost",
                "speed",
                "stability"
            ],
            "type": "string",
            "default": "accuracy",
            "description": "accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score"
        }
    }
}

get_account

The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.

Показать схему аргументов
{
    "type": "object",
    "properties": []
}

watch_task

Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.

Показать схему аргументов
{
    "type": "object",
    "required": [
        "job_id"
    ],
    "properties": {
        "model": {
            "type": "string",
            "description": "The model to keep as the pick (default: the best-scoring model). Use the model you recommended."
        },
        "job_id": {
            "type": "string"
        }
    }
}