coyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 3 months agoMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.coexternal-linkmessage-square36fedilinkarrow-up1110arrow-down10
arrow-up1110arrow-down1external-linkMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.cocoyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 3 months agomessage-square36fedilink
minus-squareSkull giver@popplesburger.hilciferous.nllinkfedilinkEnglisharrow-up15·3 months agoI think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
minus-squareZoop@beehaw.orglinkfedilinkarrow-up2·3 months ago User-Agent: bender Disallow: /my_shiny_metal_ass Ha!
I think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
Ha!