⚖️ OpenEvals

전통적 소프트웨어에서 테스트가 중요하듯이, 평가(evals)는 LLM 애플리케이션을 프로덕션으로 가져가는 중요한 부분입니다.

OpenEvals는 LangChain이 만든 오픈소스 평가 라이브러리입니다. LLM 애플리케이션을 위한 평가의 시작점을 제공합니다.

1) 설치

Python

pip install openevals

TypeScript

npm install openevals @langchain/core

LLM-as-judge 평가자를 사용하려면 OpenAI API 키 설정:

export OPENAI_API_KEY="your_openai_api_key"

2) Quick Start

첫 번째 평가

from openevals.llm import create_llm_as_judge
from openevals.prompts import CONCISENESS_PROMPT

conciseness_evaluator = create_llm_as_judge(
    prompt=CONCISENESS_PROMPT,
    model="openai:gpt-5.4",
)

inputs = "How is the weather in San Francisco?"
outputs = "Thanks for asking! The current weather in San Francisco is sunny and 90 degrees."

eval_result = conciseness_evaluator(
    inputs=inputs,
    outputs=outputs,
)

print(eval_result)

결과:

{
    'key': 'score',
    'score': False,
    'comment': 'The output includes an unnecessary greeting ("Thanks for asking!") and extra..'
}

이것은 **참조 없는 평가자(reference-free evaluator)**입니다. 일부 평가자는 참조 출력이 필요할 수 있습니다.

3) 평가자 유형

3.1 LLM-as-Judge

가장 일반적인 평가 방식입니다. 다른 LLM을 심판으로 사용합니다.

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

correctness_evaluator = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    model="openai:gpt-5.4",
)

커스터마이징:

3.2 사전 구축된 프롬프트

품질 (Quality)

안전성 (Safety)

RAG

코드

4) RAG 평가

4.1 정확성 (Correctness)

from openevals.rag import create_rag_correctness_evaluator

evaluator = create_rag_correctness_evaluator(
    model="openai:gpt-5.4",
)

result = evaluator(
    inputs="What is the capital of France?",
    outputs="Paris",
    reference_outputs="Paris is the capital of France.",
)

4.2 근거 기반성 (Groundedness)

from openevals.rag import create_rag_groundedness_evaluator

evaluator = create_rag_groundedness_evaluator(
    model="openai:gpt-5.4",
)

result = evaluator(
    inputs="What is the capital of France?",
    outputs="Paris",
    context=["Paris is the capital of France."],
)

4.3 검색 관련성 (Retrieval Relevance)

from openevals.rag import create_rag_retrieval_relevance_evaluator

evaluator = create_rag_retrieval_relevance_evaluator(
    model="openai:gpt-5.4",
)

result = evaluator(
    inputs="What is the capital of France?",
    context=["Paris is the capital of France.", "Lyon is a city in France."],
)

5) 코드 평가

5.1 Pyright (Python)

from openevals.code import create_pyright_evaluator

evaluator = create_pyright_evaluator()

result = evaluator(
    outputs="""
def greet(name: str) -> int:
    return f"Hello, {name}!"
""",
)

5.2 Mypy (Python)

from openevals.code import create_mypy_evaluator

evaluator = create_mypy_evaluator()

result = evaluator(
    outputs="""
def greet(name: str) -> int:
    return f"Hello, {name}!"
""",
)

5.3 TypeScript 타입 검사

import { createTypeScriptTypeCheckEvaluator } from "openevals";

const evaluator = createTypeScriptTypeCheckEvaluator();

const result = await evaluator({
  outputs: `
function greet(name: string): number {
  return \`Hello, \${name}!\`;
}
`,
});

5.4 샌드박스 실행

from openevals.code import create_sandbox_execution_evaluator

evaluator = create_sandbox_execution_evaluator(
    language="python",
    test_cases=[
        {"input": "2 + 2", "expected_output": "4"},
        {"input": "3 * 3", "expected_output": "9"},
    ],
)

result = evaluator(
    outputs="""
def calculate(expression):
    return eval(expression)
""",
)

6) 에이전트 궤적 평가

6.1 궤적 매치

from openevals.trajectory import create_trajectory_match_evaluator

evaluator = create_trajectory_match_evaluator(
    trajectory=[
        {"tool": "search", "input": "weather"},
        {"tool": "weather_api", "input": "San Francisco"},
    ],
)

result = evaluator(
    outputs=[
        {"tool": "search", "input": "weather"},
        {"tool": "weather_api", "input": "San Francisco"},
    ],
)

매치 모드:

6.2 궤적 LLM-as-Judge

from openevals.trajectory import create_trajectory_llm_as_judge_evaluator

evaluator = create_trajectory_llm_as_judge_evaluator(
    model="openai:gpt-5.4",
)

result = evaluator(
    inputs="What is the weather in San Francisco?",
    outputs=[
        {"tool": "search", "input": "weather"},
        {"tool": "weather_api", "input": "San Francisco"},
    ],
)

7) 문자열 평가

7.1 정확한 매치

from openevals.string import create_exact_match_evaluator

evaluator = create_exact_match_evaluator()

result = evaluator(
    outputs="Paris",
    reference_outputs="Paris",
)

7.2 Levenshtein 거리

from openevals.string import create_levenshtein_evaluator

evaluator = create_levenshtein_evaluator(
    threshold=0.8,
)

result = evaluator(
    outputs="Paris, France",
    reference_outputs="Paris",
)

7.3 임베딩 유사도

from openevals.string import create_embedding_similarity_evaluator

evaluator = create_embedding_similarity_evaluator(
    model="openai:text-embedding-3-small",
    threshold=0.8,
)

result = evaluator(
    outputs="The capital of France is Paris.",
    reference_outputs="Paris is France's capital.",
)

8) 구조화된 출력 평가

8.1 정확한 매치

from openevals.extraction import create_exact_match_evaluator

evaluator = create_exact_match_evaluator()

result = evaluator(
    outputs={"name": "Paris", "country": "France"},
    reference_outputs={"name": "Paris", "country": "France"},
)

8.2 LLM-as-Judge

from openevals.extraction import create_llm_as_judge_evaluator

evaluator = create_llm_as_judge_evaluator(
    model="openai:gpt-5.4",
)

result = evaluator(
    inputs="Extract the person's name and age.",
    outputs={"name": "John", "age": 30},
    reference_outputs={"name": "John Doe", "age": 30},
)

9) 멀티턴 시뮬레이션

9.1 사용자 시뮬레이션

from openevals.simulation import create_simulated_user

simulated_user = create_simulated_user(
    system_prompt="You are a helpful user testing a weather app.",
    model="openai:gpt-5.4",
)

result = simulated_user.invoke(
    {"messages": [{"role": "assistant", "content": "Hello! How can I help you?"}]},
)

9.2 멀티턴 시뮬레이션 with LangGraph

from openevals.simulation import create_multiturn_simulation

simulation = create_multiturn_simulation(
    app=your_app,
    simulated_user=simulated_user,
    max_turns=10,
)

result = simulation.invoke(
    {"messages": []},
)

10) LangSmith 통합

Pytest

import pytest
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

@pytest.fixture
def correctness_evaluator():
    return create_llm_as_judge(
        prompt=CORRECTNESS_PROMPT,
        model="openai:gpt-5.4",
    )

def test_correctness(correctness_evaluator):
    result = correctness_evaluator(
        inputs="What is 2 + 2?",
        outputs="4",
        reference_outputs="4",
    )
    assert result["score"]

Evaluate

from langsmith import Client
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

client = Client()

evaluator = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    model="openai:gpt-5.4",
)

results = client.evaluate(
    evaluator,
    data="dataset_name",
    experiment_prefix="my_experiment",
)

11) 커스텀 평가자 만들기

평가자 인터페이스

from typing import TypedDict

class EvaluatorResult(TypedDict):
    key: str
    score: bool | float
    comment: str | None

def my_evaluator(
    inputs: str,
    outputs: str,
    reference_outputs: str | None = None,
) -> EvaluatorResult:
    # 평가 로직
    return {
        "key": "my_eval",
        "score": True,
        "comment": "Looks good!",
    }

LangSmith 로깅

from langsmith import traceable

@traceable
def my_evaluator(
    inputs: str,
    outputs: str,
) -> EvaluatorResult:
    return {
        "key": "my_eval",
        "score": True,
        "comment": "Looks good!",
    }

12) Python async 지원

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

correctness_evaluator = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    model="openai:gpt-5.4",
)

async def evaluate():
    result = await correctness_evaluator.ainvoke(
        inputs="What is 2 + 2?",
        outputs="4",
    )
    return result

마치며: LLM 애플리케이션의 품질 보증

OpenEvals는 LLM 애플리케이션 평가의 시작점을 제공합니다.

“전통적 소프트웨어의 테스트처럼, LLM 애플리케이션에도 평가가 필요하다.”

특히 인상적인 점:

1. 다양한 평가자: LLM-as-judge, RAG, 코드, 에이전트 궤적
2. 사전 구축된 프롬프트: 품질, 안전성, RAG, 코드
3. Python + TypeScript: 두 언어 모두 지원
4. LangSmith 통합: 실험 추적
5. 멀티턴 시뮬레이션: 사용자 행동 시뮬레이션
6. 커스터마이징: 프롬프트, 모델, 출력 스키마

LLM 애플리케이션을 프로덕션으로 가져가기 위해 평가는 필수입니다. OpenEvals로 시작하세요.

🔗 관련 정보

• GitHub: https://github.com/langchain-ai/openevals
• 에이전트 평가: https://github.com/langchain-ai/agentevals
• LangChain: https://python.langchain.com/
• LangSmith: https://docs.smith.langchain.com/
• 평가 개념: https://docs.langchain.com/langsmith/evaluation-concepts