色情网站的 robots 玩法
robots.txt文件对于网站的seo的优化来讲尤为重要,下面来看看世界成人网站pornhub 的robots文件配置
User-agent: * Sitemap: https://www.pornhub.com/sitemaps.xml Disallow: /*7urren* Disallow: /*cc=* Disallow: /*correction=* Disallow: /*embed_player* Disallow: /*exclude_category=* Disallow: /*hd=1* Disallow: /*gifs/search*o= Disallow: /*incategories*p=* Disallow: */incategories*t=* Disallow: /*login=* Disallow: /*max_duration=* Disallow: /*min_duration=* Disallow: /*orientation=* Disallow: /*page*page* Disallow: /*pkey=* Disallow: /*player.php Disallow: /*player_v1.6b.swf Disallow: */pornstars/search* Disallow: /*redirect=* Disallow: /*redirectTo=* Disallow: /*redirectto=* Disallow: /*search*filter_category=* Disallow: /*search*filter_duration=* Disallow: /*search*o=* Disallow: /*o=*search* Disallow: /*showDuration=* Disallow: /*video_related* Disallow: /*watch_player* Disallow: /album/ Disallow: /albums/*-* Disallow: /albums/*/* Disallow: /albums/private* Disallow: /albums*o=* Disallow: /albums*page=* Disallow: /channels/search* Disallow: /chat/index* Disallow: /chronopop/* Disallow: /embedgif/ Disallow: /front/ Disallow: /gay/gifs/search Disallow: /gifgenerator?viewkey=* Disallow: /iframe.php* Disallow: /live/* Disallow: /live*l=* Disallow: /live*f=* Disallow: /live*n=* Disallow: /live*r=* Disallow: /live*c=* Disallow: /live*streamate=* Disallow: /live*page=* Disallow: /model/*/* Disallow: /photo/ Disallow: /phubapi* Disallow: /players/ Disallow: /playlist/rate* Disallow: /pornstars*=*=* Disallow: /pornstars*age=* Disallow: /pornstars*cup=* Disallow: /pornstar/*/playlists* Disallow: /pornstar/*/gifs* Disallow: /pornstar/*/similar* Disallow: /pornstar/*/costars* Disallow: /pornstar/*/stream* Disallow: /pornstar/*/*subscription* Disallow: /pornstar/*/videos/paid Disallow: /rlock/ Disallow: /select/video/* Disallow: /shemale/gifs/search Disallow: /svvt* Disallow: /transgender/gifs/search Disallow: /*token=* Disallow: /uploading/gifgenerator* Disallow: /user/ Disallow: /users/ Disallow: /video/ajax_search_related* Disallow: /video/ajax_video_related* Disallow: /video/player_related_datas* Disallow: /video/pornstar_suggestion_list* Disallow: /video/privates* Disallow: /video/propose_pornstar* Disallow: /video/rate* Disallow: /video/search_autocomplete* Disallow: /video/search*p=*o=* Disallow: /wall_player_v2.php Disallow: /webmasters* Disallow: /xmoov_flv/player/index.php Disallow: /video/search?search=*the+hun+ Disallow: /video/search?search=*the+hun$ Disallow: /video/search?search=*thehun+ Disallow: /video/search?search=*thehun$ Disallow: /model/*/videos/paid Disallow: */iframe-mobile* Disallow: *ajax=1* Disallow: *hd=01* Disallow: *premium=1* Disallow: *token=* Allow: /model/*/videos* User-Agent: bingbot User-agent: YandexBot User-agent: YandexVideo Allow: /embed* User-agent: Twitterbot Disallow: