https://techgazette.com.br/learn/evals/ New guides on evals, by the people who build it. Measuring what a model or agent does. en 180 Fri, 16 Oct 2026 20:48:00 GMT https://aws.amazon.com/blogs/machine-learning/building-trust-in-ai-agents-a-production-ready-evaluation-framework-with-amazon-bedrock-agentcore/ https://aws.amazon.com/blogs/machine-learning/building-trust-in-ai-agents-a-production-ready-evaluation-framework-with-amazon-bedrock-agentcore/ Fri, 16 Oct 2026 20:48:00 GMT AWS Machine Learning Evals AWS Machine Learning — Autonomous AI agents fail softly: latency stays flat while answer quality quietly erodes. This post shares a nine-stage evaluation framework, built on Amazon Bedrock AgentCore, that catches silent quality regressions acr… https://aws.amazon.com/blogs/migration-and-modernization/how-to-plan-your-migration-to-the-aws-region-in-saudi-arabia/ https://aws.amazon.com/blogs/migration-and-modernization/how-to-plan-your-migration-to-the-aws-region-in-saudi-arabia/ Thu, 15 Oct 2026 14:22:00 GMT AWS Migration & Modernization Evals AWS Migration & Modernization — The first AWS Region in Saudi Arabia is on track for December 2026. Learn which workloads to evaluate, what data to prepare, and how an AWS Optimization and Licensing Assessment helps you compare costs before you commit. https://aws.amazon.com/blogs/machine-learning/deploy-and-evaluate-models-with-amazon-sagemaker-jumpstart/ https://aws.amazon.com/blogs/machine-learning/deploy-and-evaluate-models-with-amazon-sagemaker-jumpstart/ Wed, 14 Oct 2026 13:14:00 GMT AWS Machine Learning Evals AWS Machine Learning — Learn how to deploy an available model from Amazon SageMaker JumpStart, run a lightweight and repeatable performance test using the SageMaker Python SDK, and delete the endpoint when testing is complete. https://arize.com/blog/eu-ai-act-compliance-ai-agents/ https://arize.com/blog/eu-ai-act-compliance-ai-agents/ Tue, 13 Oct 2026 16:00:00 GMT Arize Evals Arize — Learn how agent traces, human oversight, production evaluations, and log retention support EU AI Act compliance with a practical credit-assistant example. The post EU AI Act compliance for AI agents: a practical engineer… https://aws.amazon.com/blogs/machine-learning/build-long-horizon-agents-with-grok-4-7-on-amazon-bedrock/ https://aws.amazon.com/blogs/machine-learning/build-long-horizon-agents-with-grok-4-7-on-amazon-bedrock/ Tue, 13 Oct 2026 15:52:00 GMT AWS Machine Learning Evals AWS Machine Learning — Grok 4.7 is now available on Amazon Bedrock. In this post, we build a tool-calling agent on Grok 4.7 and evaluate it on three reasoning tasks at four reasoning effort levels, measuring correctness and token cost per comp… https://wf.coreweave.com/blog/tutorial-benchmarking-gpt-6-astra-vs-claude-fable-5-1-vs-gpt-5-6-sol-using-w-b-weave https://wf.coreweave.com/blog/tutorial-benchmarking-gpt-6-astra-vs-claude-fable-5-1-vs-gpt-5-6-sol-using-w-b-weave Tue, 13 Oct 2026 14:47:00 GMT CoreWeave Evals CoreWeave — Learn how to benchmark GPT-6 Astra, Claude Fable 5.1, and GPT-5.6 Sol with Weave, comparing model quality, latency, cost, and performance across practical evaluation tasks. https://devblogs.microsoft.com/foundry/more-than-a-schema-content-understanding-prebuilt-analyzers/ https://devblogs.microsoft.com/foundry/more-than-a-schema-content-understanding-prebuilt-analyzers/ Tue, 13 Oct 2026 14:20:00 GMT Microsoft Foundry Evals Microsoft Foundry — Learn how Azure Content Understanding prebuilt analyzers combine domain knowledge with grounding and confidence, and what Advanced Contextualization preview benchmarks show about quality, modeled cost, and latency. The p… https://arize.com/blog/how-to-test-agent-tool-descriptions-with-harbor-and-phoenix/ https://arize.com/blog/how-to-test-agent-tool-descriptions-with-harbor-and-phoenix/ Mon, 12 Oct 2026 14:30:00 GMT Arize Evals Arize — Build a reproducible benchmark, compare tool use across agents, and test whether a description change reduces unnecessary work while preserving outcomes. The post How to test agent tool descriptions with Harbor and Phoen… https://arize.com/blog/alignment-is-a-human-ai-workflow-custom-views/ https://arize.com/blog/alignment-is-a-human-ai-workflow-custom-views/ Fri, 09 Oct 2026 18:25:00 GMT Arize Evals Arize — Automated evals can score every agent session, but they do not automatically reflect what your team considers a good outcome. That takes human judgment. The post Alignment is a human-AI workflow: how Custom Views make ag… https://aws.amazon.com/blogs/machine-learning/icymi-what-landed-for-ai-builders-in-september-2026/ https://aws.amazon.com/blogs/machine-learning/icymi-what-landed-for-ai-builders-in-september-2026/ Fri, 09 Oct 2026 15:38:00 GMT AWS Machine Learning Evals AWS Machine Learning — A monthly recap of the latest Amazon Bedrock, Amazon Bedrock AgentCore, and Strands updates from September 2026: broader model choice, faster serverless agents with built-in evaluation, and automated knowledge base synci… https://www.digitalocean.com/community/tutorials/game-based-benchmarks https://www.digitalocean.com/community/tutorials/game-based-benchmarks Fri, 09 Oct 2026 12:00:00 GMT DigitalOcean Tutorials Evals DigitalOcean Tutorials — What game-based LLM benchmarks measure, where they work and where they fall short, how models differ in games like poker and crisis simulations, and how to use those behavioral results alongside capability benchmarks whe… https://www.anaconda.com/blog/nist-ai-600-1-generative-ai-profile-guide https://www.anaconda.com/blog/nist-ai-600-1-generative-ai-profile-guide Thu, 08 Oct 2026 16:19:00 GMT Anaconda Evals Anaconda — A practical guide to NIST AI 600-1, the Generative AI Profile: the 12 risks it names, the actions it suggests, and how to put red teaming to work. The post How to Manage Generative AI Risk With NIST AI 600-1, in Six Step… https://expo.dev/blog/your-skills-need-an-evaluation-mechanism https://expo.dev/blog/your-skills-need-an-evaluation-mechanism Thu, 08 Oct 2026 13:30:00 GMT Expo Evals Expo — How Expo tests whether coding agents actually find and use its skills, with an eval harness in CI and lessons from catalog truncation. https://www.digitalocean.com/community/tutorials/agent-observability-evaluation-signals https://www.digitalocean.com/community/tutorials/agent-observability-evaluation-signals Thu, 08 Oct 2026 00:00:00 GMT DigitalOcean Tutorials Evals DigitalOcean Tutorials — Signals flags the AI agent conversations that went wrong. It watches only the agents you turn it on for, hides personal details, keeps the data in your region, and never uses it to train AI for anyone else. It found all… https://stackoverflow.blog/2026/10/07/part-4-safety-and-governance-for-llm-systems-guardrails-pii-audit-and-memory/ https://stackoverflow.blog/2026/10/07/part-4-safety-and-governance-for-llm-systems-guardrails-pii-audit-and-memory/ Wed, 07 Oct 2026 20:48:00 GMT Stack Overflow Blog Evals Stack Overflow Blog https://stackoverflow.blog/2026/10/07/evals-as-a-deployment-gate-and-how-to-know-when-they-drift/ https://stackoverflow.blog/2026/10/07/evals-as-a-deployment-gate-and-how-to-know-when-they-drift/ Wed, 07 Oct 2026 20:31:00 GMT Stack Overflow Blog Evals Stack Overflow Blog https://devblogs.microsoft.com/foundry/choosing-azure-document-intelligence-and-content-understanding/ https://devblogs.microsoft.com/foundry/choosing-azure-document-intelligence-and-content-understanding/ Wed, 07 Oct 2026 18:45:00 GMT Microsoft Foundry Evals Microsoft Foundry — Compare Azure Document Intelligence and Azure Content Understanding across extraction architectures, workload fit, Layout capabilities, deployment options, and practical evaluation criteria. The post Azure Document Intel… https://earthly.dev/blog/ai-code-reviewers-vs-guardrail-engines/ https://earthly.dev/blog/ai-code-reviewers-vs-guardrail-engines/ Wed, 07 Oct 2026 04:00:00 GMT Earthly Evals Earthly — Your AI reviewer caught it. Next week, the same violation shows up in another repo. Across different tech stacks and custom workflows, how do you know the standard is being checked everywhere? Explore where AI reviewers… https://www.datadoghq.com/blog/how-we-replaced-our-host-vulnerability-scanner-with-the-datadog-agent/ https://www.datadoghq.com/blog/how-we-replaced-our-host-vulnerability-scanner-with-the-datadog-agent/ Wed, 07 Oct 2026 00:00:00 GMT Datadog Evals Datadog — Learn how Datadog evaluated Agent-based host vulnerability scanning, compared detection results, and preserved coverage and audit reporting during the migration. https://aws.amazon.com/blogs/mt/control-investigation-costs-before-they-start-with-aws-devops-agent/ https://aws.amazon.com/blogs/mt/control-investigation-costs-before-they-start-with-aws-devops-agent/ Tue, 06 Oct 2026 14:22:00 GMT AWS Cloud Operations Evals AWS Cloud Operations — Introduction Before an AWS DevOps Agent investigation runs its first billable query, you can see what it will cost and decide whether to proceed. The Investigation Cost Guardrail skill brings cost awareness to AWS DevOps… https://arize.com/blog/agent-traces-without-llm-judge/ https://arize.com/blog/agent-traces-without-llm-judge/ Tue, 06 Oct 2026 14:00:00 GMT Arize Evals Arize — Some agent failures are already in the trace. tracelint proves them on Phoenix OpenInference spans and fails CI, without an LLM judge. The post What agent traces can tell you without an LLM judge appeared first on Arize… https://openai.com/index/advancing-computer-use-with-ironclad https://openai.com/index/advancing-computer-use-with-ironclad Tue, 06 Oct 2026 10:00:00 GMT OpenAI Evals OpenAI — Learn how OpenAI and Ironclad are training and evaluating AI agents on complex contracting workflows to advance computer use for professional work. https://aws.amazon.com/blogs/machine-learning/evaluating-multi-agent-systems-for-explainability-and-helpfulness-with-amazon-bedrock-agentcore/ https://aws.amazon.com/blogs/machine-learning/evaluating-multi-agent-systems-for-explainability-and-helpfulness-with-amazon-bedrock-agentcore/ Mon, 05 Oct 2026 15:50:00 GMT AWS Machine Learning Evals AWS Machine Learning — Multi-agent systems need deeper guarantees than fluent responses: they must select the right tools, respect constraints, and explain their decisions. Learn how to build a Strands-based multi-agent supply chain decisionin… https://arize.com/blog/ai-agent-benchmarking-harbor-arize-phoenix/ https://arize.com/blog/ai-agent-benchmarking-harbor-arize-phoenix/ Mon, 05 Oct 2026 14:19:00 GMT Arize Evals Arize — See how Arize uses Harbor and Phoenix to benchmark AI agents, MCP servers, skills, and CLIs in reproducible sandboxes, then inspect traces and annotations. The post How we benchmark AI agents and tools with Harbor and Ar… https://www.anaconda.com/blog/indirect-prompt-injection-ai-agents https://www.anaconda.com/blog/indirect-prompt-injection-ai-agents Mon, 05 Oct 2026 14:16:00 GMT Anaconda Evals Anaconda — A hidden line of text can trick an AI agent into leaking data or sending emails. Learn how indirect prompt injection works and how guardrails stop it in time. The post The Hidden Line in the Footer: How to Stop AI Agents… https://claude.com/resources/articles/how-cresta-turned-cx-expertise-into-an-agent-builder-on-the-claude-agent-sdk https://claude.com/resources/articles/how-cresta-turned-cx-expertise-into-an-agent-builder-on-the-claude-agent-sdk Mon, 05 Oct 2026 00:00:00 GMT Claude Blog Evals Claude Blog — See how Cresta built Conductor, an agent that builds other agents, on the Claude Agent SDK, and how the team evaluates it with every new Claude model. https://blog.janestreet.com/scaling-and-benchmarking-a-critical-message-bus/ https://blog.janestreet.com/scaling-and-benchmarking-a-critical-message-bus/ Mon, 05 Oct 2026 00:00:00 GMT Jane Street Tech Blog Evals Jane Street Tech Blog — The following is part of a series of posts about 2026 summer intern projects—for more, see “What the interns have wrought, special jumbo 2026 edition” https://launchdarkly.com/blog/ai-model-management/ https://launchdarkly.com/blog/ai-model-management/ Sat, 03 Oct 2026 13:55:00 GMT LaunchDarkly Evals LaunchDarkly — Manage AI models, prompts, evaluations, rollouts, and production performance with a practical AI model management workflow. https://arize.com/blog/prompt-caching-benchmark/ https://arize.com/blog/prompt-caching-benchmark/ Fri, 02 Oct 2026 18:01:00 GMT Arize Evals Arize — We benchmarked prompt caching across DeepSeek, GLM, GPT, and Claude using Harbor evals and Phoenix traces, so we could compare cache reuse, estimated cost, and latency on the same multi-turn shopping assistant agent. The… https://wf.coreweave.com/blog/why-open-models-can-win-on-enterprise-cost-and-control https://wf.coreweave.com/blog/why-open-models-can-win-on-enterprise-cost-and-control Fri, 02 Oct 2026 14:12:00 GMT CoreWeave Evals CoreWeave — See how enterprise teams cut inference costs and keep control of fine-tuning, distillation, and guardrails by running open models on their own infrastructure. https://arize.com/blog/claude-hillclimb-production-traces/ https://arize.com/blog/claude-hillclimb-production-traces/ Fri, 02 Oct 2026 13:30:00 GMT Arize Evals Arize — Anthropic's Claude hillclimb loop designs evals and improves agents one change at a time. Here's how to run that loop against production traces in Arize AX. The post Claude’s hillclimb loop for AI agents: start with prod… https://wf.coreweave.com/blog/agentic-workflows-in-production-mechanisms-trade-offs-costs-and-design-patterns https://wf.coreweave.com/blog/agentic-workflows-in-production-mechanisms-trade-offs-costs-and-design-patterns Thu, 01 Oct 2026 17:41:00 GMT CoreWeave Evals CoreWeave — Explore how production agentic workflows are designed, orchestrated, evaluated, and governed, with practical guidance on patterns, trade-offs, costs, and failure modes. https://www.lancedb.com/blog/how-jev-compares-to-other-rerankers https://www.lancedb.com/blog/how-jev-compares-to-other-rerankers Thu, 01 Oct 2026 03:58:00 GMT LanceDB Evals LanceDB — See how Jev performs in a benchmark of 19 reranker configurations across five datasets, covering accuracy, latency, and prompt-driven relevance for LanceDB search. https://mariadb.org/mariadb-foundation-why-test-case-tags-matter-in-taf/ https://mariadb.org/mariadb-foundation-why-test-case-tags-matter-in-taf/ Wed, 30 Sep 2026 16:39:00 GMT MariaDB Foundation Evals MariaDB Foundation — In performance testing, numbers alone rarely tell the full story. A benchmark result without context is just a result floating in space disconnected from the intent, and the conditions that produced it. … Continue readin… https://www.warp.dev/blog/software-factory-as-code https://www.warp.dev/blog/software-factory-as-code Wed, 30 Sep 2026 16:00:00 GMT Warp Evals Warp — We define every agent in our software factory as code: orchestration, secrets, MCP servers, automations, runners, scorers, and benchmarks. Here’s a walkthrough of each part of the factory config. https://arize.com/blog/using-arize-ax-with-mongodb-atlas-agent-engine/ https://arize.com/blog/using-arize-ax-with-mongodb-atlas-agent-engine/ Wed, 30 Sep 2026 15:00:00 GMT Arize Evals Arize — Agents built and deployed on MongoDB’s Atlas Agent Engine can export OpenTelemetry traces to Arize AX, where teams can inspect each run, evaluate both the outcome and execution path, and test fixes before redeploying. Th… https://www.elastic.co/search-labs/blog/ecommerce-search-reranking-llm-alternative-jev https://www.elastic.co/search-labs/blog/ecommerce-search-reranking-llm-alternative-jev Wed, 30 Sep 2026 00:00:00 GMT Elasticsearch Labs Evals Elasticsearch Labs — We had Jev score Elasticsearch hybrid search results and let a short Python policy do the reranking, taking nDCG@10 from 0.9351 to 0.9565 on 250 Amazon Shopping Queries, and the code is all here. https://live-comet-marketing-site.pantheonsite.io/blog/jev-model-vs-llm-as-a-judge/ https://live-comet-marketing-site.pantheonsite.io/blog/jev-model-vs-llm-as-a-judge/ Tue, 29 Sep 2026 22:11:00 GMT Comet Evals Comet — A model that answers yes/no questions instead of writing text came out 3.5× cheaper and 3.8× faster than our gpt-4o-mini judge on 1,000 real production turns, and agreed with it on 897 of them. Here’s the test that showe… https://auth0.com/blog/integrating-claude-code-with-auth0-apis/ https://auth0.com/blog/integrating-claude-code-with-auth0-apis/ Tue, 29 Sep 2026 00:00:00 GMT Auth0 Evals Auth0 — Pair Claude Code with the Auth0 MCP Server to audit JWT configs, scaffold M2M apps, and refactor auth code from the terminal — with scoped access and security guardrails. https://stackoverflow.blog/2026/09/28/why-model-versioning-is-not-enough-for-production-ai/ https://stackoverflow.blog/2026/09/28/why-model-versioning-is-not-enough-for-production-ai/ Mon, 28 Sep 2026 18:24:00 GMT Stack Overflow Blog Evals Stack Overflow Blog — An MLOps workflow for evaluating deploying and rolling back AI applications https://clickhouse.com/blog/can-your-postgres-survive-a-bad-query https://clickhouse.com/blog/can-your-postgres-survive-a-bad-query Mon, 28 Sep 2026 15:37:00 GMT ClickHouse Evals ClickHouse — How do Postgres providers handle a query that exhausts memory? A recursive query benchmark compares query failures and cluster survival across ClickHouse Managed Postgres, Cloud SQL, PlanetScale, and Amazon RDS. https://aws.amazon.com/blogs/machine-learning/narrateai-production-ready-llm-quality-assurance-on-amazon-bedrock/ https://aws.amazon.com/blogs/machine-learning/narrateai-production-ready-llm-quality-assurance-on-amazon-bedrock/ Fri, 25 Sep 2026 16:15:00 GMT AWS Machine Learning Evals AWS Machine Learning — NarrateAI delivers production-ready LLM quality assurance on Amazon Bedrock. This post details five techniques—adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming evaluation, composit… https://inside.java/2026/09/25/limitations-of-running-a-workload-generator-in-the-same-jvm/ https://inside.java/2026/09/25/limitations-of-running-a-workload-generator-in-the-same-jvm/ Fri, 25 Sep 2026 00:00:00 GMT Inside Java Evals Inside Java — When you evaluate a system with a garbage collector and want to understand its tail latency, you often see SPECjbb2015 or DaCapo used. While both offer corrective measures for coordinated omission, only SPECjbb2015 can r… https://arize.com/blog/jev-remote-evaluator/ https://arize.com/blog/jev-remote-evaluator/ Thu, 24 Sep 2026 18:38:00 GMT Arize Evals Arize — Evaluate agent responses with Jev and Arize AX. Build a FastAPI remote evaluator, return labels and scores, and choose a threshold for your data. The post How to build a remote evaluator in Arize AX with Jev appeared fir… https://grafana.com/blog/what-if-your-agent-s-hallucinations-had-a-budget-how-to-start-using-slos-for-agent-behavior/ https://grafana.com/blog/what-if-your-agent-s-hallucinations-had-a-budget-how-to-start-using-slos-for-agent-behavior/ Thu, 24 Sep 2026 17:14:00 GMT Grafana Labs Evals Grafana Labs — At Grafana Labs, observability is what we do. So as we started building AI agents, we naturally reached for the same instincts we bring to every system: measure it, set targets, and make reliability something you can rea… https://www.elastic.co/blog/atomic-claim-checking-against-llm-hallucinations https://www.elastic.co/blog/atomic-claim-checking-against-llm-hallucinations Thu, 24 Sep 2026 00:00:00 GMT Elastic Evals Elastic — Hallucinations kept an LLM out of our knowledge base cleanup for years. Splitting generation from atomic claim verification fixed it and cleared a four-year backlog of duplicate articles. https://www.datadoghq.com/blog/jev-evals-agent-observability/ https://www.datadoghq.com/blog/jev-evals-agent-observability/ Thu, 24 Sep 2026 00:00:00 GMT Datadog Evals Datadog — See how we wired Jev into online evals on live spans and offline evals inside Datadog experiments, using a single rubric for both. https://www.twilio.com/en-us/blog/developers/tutorials/integrations/test-voice-agent-cekura-conversation-relay https://www.twilio.com/en-us/blog/developers/tutorials/integrations/test-voice-agent-cekura-conversation-relay Thu, 24 Sep 2026 00:00:00 GMT Twilio Evals Twilio — Build and test an AI voice agent using Twilio Conversation Relay and Cekura. Write metrics, run evaluations, and fix prompt bugs. https://pydantic.dev/articles/jev-evals https://pydantic.dev/articles/jev-evals Wed, 23 Sep 2026 16:00:00 GMT Pydantic Evals Pydantic — Use Jev for categorical judgments, probabilities, and numeric rubric scores in Pydantic Evals, then track the results in Logfire without a per-score fee. https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/ https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/ Wed, 23 Sep 2026 16:00:00 GMT NVIDIA Developer Evals NVIDIA Developer — An AI coding agent’s patch can pass tests yet fail when the server loads a real model and handles requests. Evaluating changes to inference-serving software...