2 pointsby shsethi4 months ago1 comment
  • genewitch4 months ago
    I think there will be some argument in the comments about whether any LLM can be considered unusable for this test.

    That is, is there any current LLM where you could prompt it with something like an interview question and it would give you perfect code one shot or zero shot?

    It might be better to host an LLM and craft the questions such that you know that the LLM will screw it up. Or, since you're self-hosting, you can have the system prompt, say insert subtle bugs that aren't syntax errors. I'll have to test if that actually does anything.

    • shsethi4 months ago
      right, getting incorrect output from an LLM deterministically might be challenging