May 24, 2024 • 2 min read
Highlight Pod #12: Traceloop Co-Founder Nir Gazit
Traceloop: Observability for LLM Applications
As more companies build production applications powered by large language models (LLMs), they are encountering challenges around monitoring and evaluating the quality of these AI model outputs.
OpenLLMetry
Traceloop has created an open source project called OpenLLMetry that provides instrumentation to capture prompts, completions, and other metadata from LLMs during runtime. This data is sent to Traceloop's observability platform, which offers a suite of metrics to evaluate output quality aspects like relevance, repetitiveness, safety violations, and more. The platform allows narrowing in on instances where models may be hallucinating or generating low-quality responses. Traceloop is also working with vendors like Microsoft and Apple to define OpenTelemetry conventions specifically for LLM observability use cases. With AI systems becoming increasingly complex and multi-modal, having purpose-built monitoring tools will be critical for responsible enterprise adoption.
