User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-agent: BLEXBot User-agent: Bytespider User-agent: CriteoBot User-agent: DataForSeoBot User-agent: DotBot User-agent: GrapeshotCrawler User-agent: linkdexbot User-agent: magpie-crawler User-agent: Mediatoolkitbot User-agent: MJ12bot User-agent: PetalBot User-agent: proximic User-agent: SemrushBot User-agent: TinyTestBot User-agent: trendictionbot User-agent: YaK User-agent: YandexBot Disallow: / User-agent: * Allow: /