They dont appreciate that specifying the details, every single detail for every possible ambiguity or choice is not something that can be done with a prompt that is a few lines or pages.
Waterfall type development processes fail for the same reason. Its just too hard to specify everything because the end product is not going to match your imagination or requirements. You might think you specified somethibg but the result will always be wrong in many ways.
Iteration, the smaller the better works because it allows one to reflect and see the outcome of a few choices. More choices means more things could be wrong and the explosion of the consequences is a difficult problem to manage.
At some point you will see a decline or even silence from your user base since the experience changed too quickly or their workflows broke. That when you'll need to cut back and argue about what really makes sense.
Since you're in a growth phase(looking at your star counts), keep going until you spot the signs I guess.