| 
	
	| 
		 
		| Option panel : Spider
 
    
 
 Accept cookiesAccept cookies generated by the remote server
 If you do not accept cookies, some "session-generated" pages will not be retrieved
 
 
 Check document typeDefine when the engine has to check document type
 The engine must know the document type, to rewrite the file types. For example, if a link called /cgi-bin/gen_image.cgi generates a gif image, the generated file will not be called "gen_image.cgi" but "gen_image.png"
 Avoid "never", because the local mirror could be bogus
 
 
 Parse java filesMust the engine parse .java files (java classes) to seek included filenames?
 It is checked by default
 
 
 SpiderMust the engine follow remote robots.txt rules when they exist?
 The default is "follow"
 
 
 Update hackAttempt to limit transfers by wrapping known bogus responses from servers.
  For example, pages with same size will be considered as "up to date", even if the timestamp seems
  different. This can be useful for many dynamically generated pages, but this can also cause
  not-updated pages in rare cases.
 
 
 Tolerant requestsTolerate wrong file size, and make requests compliant with old servers
 It is unchecked by default, because this option can cause files to become bogus
 
 
 Force old HTTP/1.0 requestsThis option forces the engine to use HTTP/1.0 requests, and avoid HEAD requests.
 Useful for some sites with old server versions, or with many dynamically generated pages.
 
 
 
 
 
 
 Back to Home |  |  |