Technology

From 2,500 Requests to 24: Rebuilding Star History Without Stargazers

· GlobeWire Daily Tech Desk · Source: github.blog

2 min read

Key takeawaysMaintainers look at star history to answer practical questions about project trajectories around launches or events. Historically, obtaining this curve via the GitHub API required requesting pages of individual stargazer records, keeping timestamps, and discarding identities. Over time, this design posed privacy risks from scraping and caused database…
Decorative background featuring floating green cubes, including one with the GitHub invertocat logo.

Image: Taylor Cuilty, Sarita Iyer · Photo source · Resized and letterboxed to 16:9; no AI generation

Maintainers look at star history to answer practical questions about project trajectories around launches or events. Historically, obtaining this curve via the GitHub API required requesting pages of individual stargazer records, keeping timestamps, and discarding identities. Over time, this design posed privacy risks from scraping and caused database performance issues on highly starred repositories due to expensive counting and pagination.

After restricting bulk stargazer and watcher list access to repository administrators and collaborators to mitigate immediate risks, GitHub consulted with maintainers and tool builders. They discovered that tools like Star History needed aggregate counts and dates rather than individual usernames. This collaboration led to the development of a purpose-built endpoint designed to deliver historical trends efficiently without exposing user identities.

Understanding the Original Design Limitations

The original stargazer endpoint was built to list people who starred a repository rather than to serve as a historical event log. Reconstructing growth curves meant retrieving thousands of individual identities and processing every current star. Pagination bugs further exacerbated database load, causing thousands of slow queries and timeouts.

Designing an Efficient Aggregate Endpoint

Initial attempts to return timestamps without identities still required pagination through thousands of individual records. Simple SQL aggregation with `LIMIT` and `OFFSET` reduced JSON payload sizes but failed to reduce expensive database grouping and sorting across millions of rows.

The final solution shifts the order of operations by filtering stars by calendar time before counting:

  • Time-Range Filtering: Requests translate page parameters into indexed `created_at` date ranges.
  • Optimized Requests: A single request now returns up to 30 weeks of daily star counts.
  • Reduced Database Load: Instead of scanning all current stars repeatedly, popular repositories like `react/react` now require approximately 24 requests instead of 2,511.

Practical Implications and Verification

The new aggregate endpoint protects user privacy while providing maintainers with predictable performance. However, scaling limitations remain for exceptionally dense weeks with unusually high star activity. Developers can verify documentation and share feedback directly through official GitHub community discussions.

Key Takeaways

  • Focus on the actual use case rather than legacy data structures.
  • Minimize data exposure by returning only what is strictly necessary.
  • Filter data before aggregation to reduce database query costs.
Share this article
Read the original source: From 2,500 requests to 24: Rebuilding star history without the stargazers

AI-assisted reporting; source claims are attributed and have not been independently verified.

About the author

GlobeWire Daily Tech Desk — AI-assisted news based on attributed published sources.

More to read