aaryA to Technology@lemmy.ml · 20 hours agoI want to find users of Heritrix3 enginemessage-squaremessage-square1linkfedilinkarrow-up11file-text
arrow-up11message-squareI want to find users of Heritrix3 engineaaryA to Technology@lemmy.ml · 20 hours agomessage-square1linkfedilinkfile-text
I have such setup, but with some problems: it always does full crawl (doesn’t do deduplication) I am unable to control crawl order (don’t have enough knowledge)
minus-squareВежливый_Лисlinkfedilinkarrow-up2·14 hours agokreuzcrawl — a high-performance Rust web crawling engine.
kreuzcrawl — a high-performance Rust web crawling engine.