Show HN: ArchGW – An open-source intelligent proxy server for prompts(github.com)
39 pointsby sparacha5 days ago6 comments
  • adilhafeez4 days ago
    Hi - this is Adil the co-founder who developed archgw. We are working tirelessly to create a framework that would help developers write agentic application without having to write all the crufty/boilerplate code. At the very minimum we provide observability and logging without adding much overhead. You can simple plug arch gateway into your existing LLM application and you'd start seeing details like time-to-first-token, total latency, token count and tons of other observability details. I do recommend start tinkering with our getting started page here [1]

    And for a bit more advanced use cases I do recommend looking at llm_routing [2] demo and currency_exchange demo [3].

    We currently support providing seamless interface to major providers like openai, mistral, deepseek and also support hooking up to local providers like ollma [4]

    [1] - https://github.com/katanemo/archgw?tab=readme-ov-file#quicks...

    [2] - https://github.com/katanemo/archgw/tree/main/demos/use_cases...

    [3] - https://github.com/katanemo/archgw/tree/main/demos/samples_p...

    [4] - https://github.com/katanemo/archgw/tree/main/demos/use_cases...

  • mikram4 days ago
    I think I saw this a few months ago, but never followed up. Why train your own models? Aren't you better off using GPT or something like that to handle the tasks Arch uses specialized models for?
    • sparacha4 days ago
      Speed. And separately, instruction fine-tuning an LLM for a specialized task like function calling or guardrails == better performance. Even Anthropic and other model providers suggest you separate tasks for LLMs to improve overall user experience

      We happen to take those tasks that are non-business or domain specific related and trained our models to offer SOTA performance for 1/10th the cost and 10x the speed. For e.g. Arch-Function can process 5k/tokens per sec

  • _nh_4 days ago
    How do you compare with https://github.com/comet-ml/opik in observability?
    • sparacha4 days ago
      Opik is an evaluation tool first. Arch is a proxy server built on top of Envoy so it borrows from a very robust observability source. They both are complimentary in many ways
      • honorable_judge3 days ago
        Envoy is compatible with OTel out of the box. That's a big plus for observability. Plus Envoy is designed for high-load dataplane (in the request path worklaods) and used in every modern stack. There are several advantages on using Arch as the source of observability (traces, metrics, logs)
  • 5 days ago
    undefined
  • sparacha5 days ago
    Woud love feedback. See if it is useful, or what adaptations would make it useful.
  • 5 days ago
    undefined