coyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 5 months agoMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.coexternal-linkmessage-square36fedilinkarrow-up1110arrow-down10
arrow-up1110arrow-down1external-linkMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.cocoyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 5 months agomessage-square36fedilink
minus-squareSkull giver@popplesburger.hilciferous.nllinkfedilinkEnglisharrow-up15·5 months agoI think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
minus-squareZoop@beehaw.orglinkfedilinkarrow-up2·5 months ago User-Agent: bender Disallow: /my_shiny_metal_ass Ha!
I think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
Ha!