robots.txt文件对于网站的seo的优化来讲尤为重要，下面来看看世界成人网站pornhub 的robots文件配置

User-agent: *
Sitemap: https://www.pornhub.com/sitemaps.xml
Disallow: /*7urren*
Disallow: /*cc=*
Disallow: /*correction=*
Disallow: /*embed_player*
Disallow: /*exclude_category=*
Disallow: /*hd=1*
Disallow: /*gifs/search*o=
Disallow: /*incategories*p=*
Disallow: */incategories*t=*
Disallow: /*login=*
Disallow: /*max_duration=*
Disallow: /*min_duration=*
Disallow: /*orientation=*
Disallow: /*page*page*
Disallow: /*pkey=*
Disallow: /*player.php
Disallow: /*player_v1.6b.swf
Disallow: */pornstars/search*
Disallow: /*redirect=*
Disallow: /*redirectTo=*
Disallow: /*redirectto=*
Disallow: /*search*filter_category=*
Disallow: /*search*filter_duration=*
Disallow: /*search*o=*
Disallow: /*o=*search*
Disallow: /*showDuration=*
Disallow: /*video_related*
Disallow: /*watch_player*
Disallow: /album/
Disallow: /albums/*-*
Disallow: /albums/*/*
Disallow: /albums/private*
Disallow: /albums*o=*
Disallow: /albums*page=*
Disallow: /channels/search*
Disallow: /chat/index*
Disallow: /chronopop/*
Disallow: /embedgif/
Disallow: /front/
Disallow: /gay/gifs/search
Disallow: /gifgenerator?viewkey=*
Disallow: /iframe.php*
Disallow: /live/*
Disallow: /live*l=*
Disallow: /live*f=*
Disallow: /live*n=*
Disallow: /live*r=*
Disallow: /live*c=*
Disallow: /live*streamate=*
Disallow: /live*page=*
Disallow: /model/*/*
Disallow: /photo/
Disallow: /phubapi*
Disallow: /players/
Disallow: /playlist/rate*
Disallow: /pornstars*=*=*
Disallow: /pornstars*age=*
Disallow: /pornstars*cup=*
Disallow: /pornstar/*/playlists*
Disallow: /pornstar/*/gifs*
Disallow: /pornstar/*/similar*
Disallow: /pornstar/*/costars*
Disallow: /pornstar/*/stream*
Disallow: /pornstar/*/*subscription*
Disallow: /pornstar/*/videos/paid
Disallow: /rlock/
Disallow: /select/video/*
Disallow: /shemale/gifs/search
Disallow: /svvt*
Disallow: /transgender/gifs/search
Disallow: /*token=*
Disallow: /uploading/gifgenerator*
Disallow: /user/
Disallow: /users/
Disallow: /video/ajax_search_related*
Disallow: /video/ajax_video_related*
Disallow: /video/player_related_datas*
Disallow: /video/pornstar_suggestion_list*
Disallow: /video/privates*
Disallow: /video/propose_pornstar*
Disallow: /video/rate*
Disallow: /video/search_autocomplete*
Disallow: /video/search*p=*o=*
Disallow: /wall_player_v2.php
Disallow: /webmasters*
Disallow: /xmoov_flv/player/index.php
Disallow: /video/search?search=*the+hun+
Disallow: /video/search?search=*the+hun$
Disallow: /video/search?search=*thehun+
Disallow: /video/search?search=*thehun$
Disallow: /model/*/videos/paid
Disallow: */iframe-mobile*
Disallow: *ajax=1*
Disallow: *hd=01*
Disallow: *premium=1*
Disallow: *token=*

Allow: /model/*/videos*


User-Agent: bingbot
User-agent: YandexBot
User-agent: YandexVideo
Allow: /embed*

User-agent: Twitterbot
Disallow: