free web crawler, if use together with Search engine builder -lexst-SEA, you can index 10 billion pages.Multi threads and distributed free web crawler, for both internet and interanet.You can control how frequency the spider should crawl your pages, you can save the pages locally or sent to a search-engine applicant.

End user license