Turns out the LangGraph code uses the official OpenAPI library which eagerly instantiates an HTTPS transport and 65% of runtime was dominated by ssl.create_default_context (SSLContext.load_verify_locations) when I tested using pyinstrument. This overhead is further exasperated by the fact that it's happening twice - once for the sync client, and a second time for the async client. Rest of the overhead seems to be Pydantic and setting up the initial state/graph.
Agno wrote their own OpenAPI wrapper and defers setting up the HTTPS transport during agent creation, so that cost still exists it's just not accounted for in this "benchmark". Agno still seems to be slightly faster when you control for this, but amortized over a couple of requests it's not even a rounding error.
I hope the developers get rid of this "claim" and focus on other merits.
Like 5s+/-1s vs 4.95s+/-1s
Appreciate if you didn’t make up stuff.
Instantiating an agent is not the bottleneck for LLMs. Two hundredths of a second is a rounding error compared to what the model costs in time.