cm0002@lemmy.world to Technology@lemmy.worldEnglish · 11 months agoOpen Source devs say AI crawlers dominate traffic, forcing blocks on entire countriesarstechnica.comexternal-linkmessage-square17linkfedilinkarrow-up1250arrow-down12cross-posted to: opensource@programming.dev
arrow-up1248arrow-down1external-linkOpen Source devs say AI crawlers dominate traffic, forcing blocks on entire countriesarstechnica.comcm0002@lemmy.world to Technology@lemmy.worldEnglish · 11 months agomessage-square17linkfedilinkcross-posted to: opensource@programming.dev
minus-squaremesamunefire@lemmy.worldlinkfedilinkEnglisharrow-up59·11 months agoYep same thing. I have some small servers and was getting hammered by openai ip controlled ai crawlers not respecting robots.txt. had to block all their IP addresses and create an AI black hole in order to stop them ddos ing my tiny site(s).
minus-squaretempest@lemmy.calinkfedilinkEnglisharrow-up33·11 months agoDid you not pay your protection money to CloudFlare?
minus-squareCosmic Cleric@lemmy.worldlinkfedilinkEnglisharrow-up28arrow-down4·11 months agoMan, this current age of AI really sucks. This comment is licensed under CC BY-NC-SA 4.0
minus-squareburghler@sh.itjust.workslinkfedilinkEnglisharrow-up2·11 months agoYou licensed your comment? You just saw first hand that AI crawlers don’t care about legal barriers.
Yep same thing. I have some small servers and was getting hammered by openai ip controlled ai crawlers not respecting robots.txt. had to block all their IP addresses and create an AI black hole in order to stop them ddos ing my tiny site(s).
Did you not pay your protection money to CloudFlare?
Man, this current age of AI really sucks.
This comment is licensed under CC BY-NC-SA 4.0
You licensed your comment? You just saw first hand that AI crawlers don’t care about legal barriers.