Skip to content

Latest commit

 

History

History
15 lines (12 loc) · 768 Bytes

README.md

File metadata and controls

15 lines (12 loc) · 768 Bytes

simple-go-crawler

This is a simple web crawler to collect all links inside a website. This crawler support Multi Page Application as well as Single Page Application.


Basic Usage

To use this app, you just need to run the server and then post your url to /api/crawl. For Single Page Applications you might need to consider html element to be visible before you start scrapping.
Change chromedp.WaitVisible('your-element') to match your requirement. It will wait until those particular element appear before start scrapping.

Dependencies

  1. chromedp see here
  2. GoFiber see here

Unit testing screenshots

CSR
SSR