2 pointsby cjjuice5 hours ago2 comments

sango_ta5 hours ago
That's kinda neat. Do you do any decoding tricks to handle varying processing capabilities across device models and ipads? Like shortening ctx length or maybe looking into the LFM models that have MoEs with like 1b active?
- cjjuice3 hours ago
  Right now I am just restricting it to devices with 8GB RAM. I plan to start testing older phones soon to see if I can get it running well on them. MoE with low active params is interesting. I wonder if there is smaller models based on specific pairs (En/Es, ect.)
robby044 hours ago
[dead]