Flexible and Scalable low-code Web Crawler. Give it a try on the LIVE playground: https://llm-web-crawler.vercel.app/
Uses BuildShip - a visual AI workflow builder to extract and gather data from your websites or sources. This data can then be used as a knowledge base to power your own LLM apps 🤖, or paired with BuildShip's AI Assistant to unlock powerful use cases and enhance your business or services.
| Node | Info | Documentation | Template |
|---|---|---|---|
| Scrape | Easy to get started with, scrape a given web URL and return the text content. Works great for less complex sites that don't rely on JavaScript to load. | Read more | Remix |
| Dynamic Scrape | Scrape a given web URL and return the text content. This method works well for more complex sites and allows for more interactive scraping by providing a set of steps to execute after loading the page. For example, loading an ecommerce site, searching for an item, and then scraping the search results info. | Read more | Remix |
| Web Crawler | Extract data from an entire website by crawling through and scraping all its pages. Perfect for aggregating data to create your own custom GPTs or "Chat with Data" apps. | Read more | Remix |
| LLM Extraction | Extract structured data (just the data you care about) from any website. No need to scrape an entire webpage; simply specify the URL and the fields you want to extract. The LLM will handle the rest, delivering only the relevant data in a structured format. | Read more | Remix |
-
First clone a template using most relevant to your using the following links. LLM Extractor
Crawler
Static Web Scraping
Dynamic Web Scraping -
Run or customize the template as per your usecase
-
Click Ship to deploy as an API or scheduled job
Read full documentation to learn more
