C-Command Software Forum

Capture from authenticated web pages via browser cookies

I encountered an interesting piece of software last year, SiteCrawler by Lighthead, and while it is a completely different kind of application, it has one method for downloading authenticated web pages that I wish was available in EagleFiler

SiteCrawler can integrate with Safari’s cookie cache. You can log in to a website via Safari (e.g. Facebook or the New York Times), then run SiteCrawler and it transparently uses Safari’s cookies while loading the pages. See their homepage section “Safari Integration.” http://lightheadsw.com/sitecrawler/

I’m not sure how complex this would be to implement in EagleFiler practice, but it seemed like the kind of elegant solution that EagleFiler would use, so I’m mentioning it here as a future feature idea / request.


To be clear, EagleFiler does access and use the persistent Safari cookies. This will work for many sites that require login, including the New York Times. Some Web sites, like Facebook, specify that their login cookies should be confined to a single browsing session, within a single application. SiteCrawler is getting at these by running a haxie inside Safari itself and then sending the cookies to the SiteCrawler application. Thus far, EagleFiler has avoided the haxie approach and relied entirely on supported interfaces for integrating with other applications.