June 22, 2025
409 views
Project Page Documentation
Define site-specific crawling settings like frequency, duration, data fields, and additional information.
| Site name | Optional label for the website (e.g., "Nike Product Page"). |
| Website URL | The actual URL of the site to crawl. |
| Data format | Specify required format (e.g., JSON, CSV, XML). |
| Crawl Frequency | Options include one_time, weekly, monthly. Affects scheduling. |
| Duration | Total duration in months for which the site should be crawled. |
| Data Points | A comma-separated list of what data to extract (e.g., name, price, image). |
| Additional Info | Free text for any extra instructions or clarification. |
Need more help? Just email us at
contact@crawlfeeds.com