We are all running lots of sessions at this point where we don't look at everything that is happening every second and if something takes an hour or an hour and 30 minutes, there is no real change. Agents can wait.
Our value added will come from more than just inference. Stay tune for more :)
Which results in the cost structure being something like a gym membership ?
ie. oversubscribed and underutilized ?
Are you going to throttle or ban the users that fill their pipelines 24/7 ?
Think about the difference of paying by the minute or text messages for cell phones in the old days compared to an all you can it (with some restrictions) that you get on cell phone plans today.
Unless users violate the ToS we do not ban 24/7 and if there is more capacity at certain times you will not get queued.
In fact, the subscription gives you priority requests to get ahead of the line for certain things and the rest will flow to the background or if you have specific workloads that are not time sensitive send them there anyway.
Think about continuous AI usage of doing things like sweeping for tickets, CRM enrichment, code reviews in the background and more.
You're going to operate it at capacity, and requests will fill a queue, with the goal being, to keep it over 100% queued.
I wish you much success.
Also, you might want another pricing model, to compliment this one, for someone like a credit card processor who wants to inspect every transaction for fraud, and latency is more important, whenever you think you can compete on that metric of course.
Volume and latency versus tokenized consumption.
It does make sense. Some people want those instead of counting pennies.
We do have the ability to provide dedicated servers and partial servers exactly for this type of needs as well as regulated industries and others.
We also have a specific plan with access to abliterated/uncensored models with a required KYC/KYB.