Within like an hour or two pg emailed me asking me to stop. I didn't know it at the time, but HN was being run on a rusty potato and scraping the homepage every 5 or 10 seconds was causing significant load.
curl -N -H "Accept: text/event-stream" "https://hacker-news.firebaseio.com/v0/maxitem.json"
The potato with multiple eyes works to serve one of the more important and trafficked properties on the Internet.
I like HN, but it's really only important within a very niche subset of the Internet, and it also doesn't have much traffic. There's like a single post submitted every two minutes. That's not much.
Nonetheless, i suspect that HN probably is quite efficient, just based on what I know about dang. Even so, the parent claim was that it was popular and important, not that it was efficient
https://www.reddit.com/r/TheoryOfReddit/comments/s2dvr4/the_...
And live firebase source/api (run by HN) used for this: https://github.com/HackerNews/API
(just making sure it's obvious - I didn't create this site! all credits to @jerbear4328 who is here on HN - I'll email them to let them know it's trending) :-)
(edit: terminal version available?)
https://github.com/ggerganov/hnterm
from the author of llama.cpp (!!).
it also has a beautiful web version thanks to emscripten: https://hnterm.ggerganov.com/
This is literally the only thing that seems missing- Stupendous work!
Edit: Through my own tool I see this comment got insta-marked as [dead], rude.
https://news.ycombinator.com/item?id=39669357
(Edit: Yep, this comment is dead on arrival.)
Separately I didn’t know posts/submissions could be instantly flagged dead and I didn’t know there are decade+ old bots still configured to spam this site https://news.ycombinator.com/submitted?id=VivaTechnics
Would be slightly more contextual if the title of the original post was displayed.
For scraping HN directly, in my experience you have to go extremely slow, like 1 minute between fetching items. And if you get blocked, it may be better to wait a long time (minutes) before trying again rather than exponential backoff, in order to get out of the penalty box. You'll need a cache for sure.
{
"by" : "jkarneges",
"id" : 45533018,
"kids" : [ 45533616 ],
"parent" : 45532549,
"text" : "The HN/Firebase API doesn't make this easy. For <a href=\"https://hnstream.com\" rel=\"nofollow\">https://hnstream.com</a> I ended up crawling items to find the article.",
"time" : 1760043552,
"type" : "comment"
}
If one were to start at the page endpoints (eg /topstories) one could add references to origin ids while preloading comments, and probably cover the most likely to be referenced ID, and even make traversal up the tree even more efficient.
I just wish there was a dark mode.. Let me install night reader to see how well it looks :p
Edit: would've loved if there was a way to sign up or make comments from there, oh well :< I wish there was, I am not sure if that's possible tho but I hope it could be.
Static hosting is super cool and I am going to tinker with it to have monospace-web instead for something like this https://owickstrom.github.io/the-monospace-web/
Thanks a lot for creating this! Appreciate it, I haven't read the source code but I am going to read it oneday (If I don't procastinate lol)!
Have a nice day!
https://news.ycombinator.com/item?id=20652157
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
But dang has said numerous times that he doesn't read everything, that would probably not even be feasible.
The question is, YC being what it is, are they using LLMs to automate moderation or do sentiment analysis?
Great potential job for an LLM to potentially spot abuse. The existing mod tools/system seem to work quite well.. I like HN how it is!
Could be a cool movie scene where the stream of data is actually live HN comments lol.
To shill my own use of the API:
I did an animated "replay" view for historical threads like Rust 1.0 launch: https://hn.unlurker.com/replay?item=9551937
And a (static, refreshable) view of recent activity grouped by topic: https://hn.unlurker.com
But anyways, just as a headsup for a moment I was confused that I wasn't getting the verification mail and I was literally going to comment it but then I got the mail just in time, so maybe to anyone out there, maybe be a bit patient as its worth it (or it could totally just be an issue from my side as I spammed it twice or maybe it got congested I am not sure)
EDIT: It wasn't that they sent me 2 mails but rather that they sent me 1 mail and I clicked on it and then got another and I thought it was because of me spamming verification twice but it turns out that they sent me a mail because someone had responded to me :)
Neato. I really like this.
claude also said "The negative comments tend to be more substantive critiques of systems or research approaches rather than personal attacks." - so... that's good? :-)
There is an API endpoint called "updates" but it's incredibly vague about what it actually shows, or how persistent it is. It might show edits.
As a side feature, that would be neat though.
Edit: Felt like it took almost a minute, I wasn't timing precisely because I didn't expect it to take so long!