[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Words, terms, tokens, etc.
On Wed, 8 Mar 2000, Jorge Stolfi wrote:
> Anyone who deals with text has the need to distinguish between those
> two concepts. Unfortunately each community (if not each author) will
> pick different names for them. For instance, I recall that Gabriel's
> Zipf law paper uses "word" for the instance, and "token" for the
> dictionary entry; whereas the compiler-writer community makes precisly
> the opposite choice.
Quite right! And *probably* :-) the compiler-writer comunity is right. I
remember that I defined the terms, though.