Git scraping: track changes over time by scraping to a Git repository
Report a link problem
Submission timeline
2007–2026One slot for every year since HN launched. Height is that year's peak points; orange marks a 100+ point or 50+ comment breakout. Select a bar to open its strongest thread.
First comments on top threads
HN comment orderOne of my favourite things about Git scraping is that, thanks to CI providers like GitHub Actions which are free for open projects, it costs nothing to run. https://github.com/simonw/ca-fires-history for example scrapes every 20 minutes and runs on the free GitHub Actions plan. I've run these on free Travis CI and Circle CI plans in the past as well. If you want to run a private Git scraper you can pay money to do so, which seems very reasonable to…
I've been promoting this idea for a few years now, and I've seen an increasing number of people put it into action. A fun way to track how people are using this is with the git-scraping topic on GitHub: https://github.com/topics/git-scraping?o=desc&s=updated That page orders repos tagged git-scraping by most-recently-updated, which shows which scrapers have run most recently. As I write this, just in the last minute repos that updated include: queensland-traffic-conditions…
> Update 5th January 2022: I released a tool called git-history that helps analyze data that has been collected using this technique. https://simonwillison.net/2021/Dec/7/git-history/
The first top-level comment from each of the four biggest threads, in HN’s own order. Excerpts are shortened; open a comment for full context.
- Breakout years
- 2
- Total points
- 643
- Total comments
- 162
100+ points or 50+ comments
reference only — not used in Hall rules or ranking
reference only — not used in Hall rules or ranking
Every submission
| Date | Title as submitted | By | Points | Comments |
|---|---|---|---|---|
| 2020-10-09 | Git scraping: track changes over time by scraping to a Git repositoryFirst breakout · Best thread | simonw | 471 | 95 |
| 2023-04-24 | Git scraping: track changes over time by scraping to a Git repository (2020) | gstipi | 5 | 1 |
| 2023-08-10 | Git scraping: track changes over time by scraping to a Git repository (2020) | stefankuehnel | 1 | 0 |
| 2023-08-10 | Git scraping: track changes over time by scraping to a Git repositoryLatest 20+ point return | ekiauhce | 166 | 66 |