I scraped nearly 1T of html-ish data from the Internet Archive version of a web site that was recently taken offline (using this fascinating tool), and the data is noisy and trashy. I intend to normalize it, clean it, de-dupe it, and transfer it into an industrial-strength database. But in the meanwhile, I’m thinking of parking it in EagleFiler, so I can grab urgent data here and there as needs arise.
I don’t have nearly enough space on my HD, so I’m thinking of storing my EF library on an external drive. But I have a distant memory of doing that once before and running into poor results, and eventually restoring it to my HD. Is that a false-memory? Is EF perfectly ok running from external?
I just noticed the pref choice of web page format, which seems (?) to default to web archive. Should I leave this alone? In most contexts, I’d prefer PDF, but I’d imagine for EF’s purposes, archive or html might work best.
It should be fine. EagleFiler was designed from the start to support external drives, since I have some large libraries that don’t fit on my internal. I don’t recall anyone ever reporting any problems with this, but please let me know if you run into any issues.
The default is “Web archive” since that’s the most lossless. But you can choose whatever you want. Personally, I’ve been using PDF lately.
Thanks, Michael. I’ll report problems for sure. My library is already pretty loaded with data - I’ve complained to you about launch latency (you said it was normal). Let’s see what happens if I add a Tb of data, plus the latency of USB plus the latency of a physical hard drive. Maybe I should spring for a SSD external, but they’re still pricey.
I’ve been using PDF lately.
Me too, just generally. I’ve finally taken a stand for the format, and have been putting anything readable into PDF and reading on mobile via Readdle Docs. Big relief to have it all in one format and one place for reading with one app. For writing, I still stick with .txt or .md (or .doc if handing in a manuscript). But I don’t see this applying to the sort of material I keep in EF, so I’ll leave on web archive default. Maybe you use EF for a broader range of materials.
If you have a large number of files, it may take a bit to open the library and read their metadata from a spinning disk. But actual browsing/viewing should be snappy.
Yes, I have tons of .txt and .md files in EagleFiler.
@Jimmbo: Is EF perfectly ok running from external?
In addition to what Michael said, I rather like the trick of having the library on the SSD but moving the files to the HDD. The EagleFiler manual notes how this is possible using symbolic links. This has the advantage of loading the index much faster while the bulk of the storage space is offloaded to the HDD.
My only concern is that in my experience symlinks are not super robust. They don’t always survive migration to new computers, shifts of directory structures, the vagaries of iCloud or DropBox, etc.
In fact, I can’t think of a single symlink solution that stood the test of time for me. It always turns out to be a short term kludge that breaks in the end.
I ran EF on a file server - so slow, I gave up EF totally once, and used only minimally later.
I’ve just got a fast Samsung X5, via Thunderbolt/ USB-C. On paper it claimed a similar spec to the internal disk. Experientially this appears true. Makes a world of difference. EF now works as well as others describe, Not v. cheap - until you compare with costs of bigger disks from apple…
Of course an SSD is best, but even a spinning hard drive should be fine, so long as it’s directly connected to the Mac. That’s what I use for my multi-TB libraries. There’s a much bigger difference between network/local than spinning/SSD.
