More about Meta-data
While I was implementing the keyword blogging system described below,
I had a revelation about meta-data. There is an awful lot of meta-data
already present on your system, and it could be a real help when searching
for things on your system. Unfortunately, it's almost never used. 
<p>
When I talk about meta-data in this entry, I'll be talking about any arbitrary
information associated with a particular file. I'm not talking about file
creation times or file permissions - those are indeed meta-data, but they
already exist for nearly every file and are already widely searchable.
<p>
The meta-data I'm talking about tends to be associated to user-created files. 
For example, the mp3s you downloaded back when Napster was new have ID3 headers
attached to the listing the artist and song. Those
pictures you downloaded off of your digital camera have meta-data on them in the
form of EXIF information. All those emails that you send and receive store
a ton of information in their headers, and that
letter you typed up in Word probably contains more information in it than
you'd like to know.<p>

<!--mention file extensions?-->

These files also tend to be bunched up together. If you run Windows, they're
likely to be in your <code>my documents</code> folder, and if you run *nix,
they're likely to be buried somewhere in your <code>/home</code> directory. 
If you run both,
they're probably scattered in various places on your hard drive. Even if
you keep them hidden in some weird place, files with meta-data tend to cluster
together.<p>
So far what I've concluded is that files with meta-data tend to be created or
intentionally downloaded by the user of a computer. This means that they are
quite likely files that the user considers important. Furthermore, they tend
to be bunched up in large numbers in a few directories. Since files with 
meta-data tend to be the files that are important to the user and bunched up
in small areas, the user would likely benefit tremendously from an improved
computer search function.
<h2>But how?</h2><p>
When I first started thinking about meta-data in general, I thought about it
on a file system level. "Wouldn't it be great", I thought, "if things like
meta-data were stored in one consistent way across the file system, so that it
stayed attached to the file? That way, searching meta-data would be a breeze!" 
So I bang around Google for a while, and find
out that HFS for Macintosh had it way back in the day, and that all kinds of
file systems have it (some in a form called extended attributes, or XATTR) 
today.<p>
However, it seems that this meta-data is of limited use. After all, not every
file system has the concept of meta-data, so when you want to send your MP3
over to your friend whose file system doesn't support your type of meta-data,
he'll lose all of that handy information. Meta-data on an operating system
level fails for anything outside of a single machine for the same reason. If
I relied on the Windows registry to tell me who sang my copy of "Stairway
To Heaven", when I reboot into Linux, that information has become instantly
worthless.<p>
Thus, meta-data seems only to make sense on "type of file" level, where it
is usually stored. Mp3s share the ID3 format, emails share RFC 822
headers, and images share the EXIF format. Each type of file has a different
type of meta-data format, because they each have very different requirements
for the information they need to store.<p>
<h2>Does this have a point?</h2><p>
There's a couple of points. First, the way that meta-data is currently stored
makes sense. File system-level meta-data is useful for attributes shared by all
files - time of creation and security information. OS-level meta-data is
useful to help the OS do its job marshaling the programs on a computer.
Information specific to a class of files is best stored inside of those files,
to facilitate their transmission and parsing by programs designed for that
class of files.<p>
Second, current computer search technologies do a woefully inadequate job of
searching the large amount of meta-data present on a modern computer system,
and an improved search would be very useful. To search a file system 
effectively, a program would have
to be able to read and understand a fairly large variety of unrelated file
formats and their associated meta-data. While tedious to write, a tool that 
moved a step
beyond simple plain-text searches of file names and contents would have a shot
at becoming an indispensable tool for desktop searching.
<!--keywords: meta-data, search, computer-->
