Fast and Elegant Scraping Framework for Gophers


Colly provides a clean interface to write any kind of crawler/scraper/spider

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.


  • Clean API
  • Fast (>1k request/sec on a single core)
  • Manages request delays and maximum concurrency per domain
  • Automatic cookie and session handling
  • Sync/async/parallel scraping
  • Distributed scraping
  • Caching
  • Automatic encoding of non-unicode responses
  • Robots.txt support
  • Google App Engine support
 1 2 3 4 5 6 7 8 91011121314
func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a", func(e *colly.HTMLElement) {

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)


Batteries included

Colly comes with all the tools you need for scraping.


Business ready

Free for commercial use.


Open Source

Development of Colly is community driven and public.