Hello,
On 2020-05-29 10:16 p.m., Yair Lenga wrote:
Wanted to suggest that the team will look (again) at implementing
--unsorted option for 'uniq'.
The idea was proposed (and rejected) about 10 years ago
(https://lists.gnu.org/archive/html/coreutils/2011-11/msg00016.html).
Lot of things have changed from the past.
[...]
Can you advise/provide feedback. I'm sure that there will be many
volunteers (me included) to contribute to such important improvement.
"uniq" is standardize by POSIX to work on "comparing adjacent lines"
(from:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/uniq.html ) -
hence the requirement to pre-sort the input.
While it could be extended with a completely different hash-based
implementation, I don't think this is likely to happen.
As an alternative (and a shameless plug), allow me to point to
GNU Datamash ( https://www.gnu.org/software/datamash/ ).
On one hand, it already has a hash-based implementation to
remove duplicated fields (called "rmdup").
consider the following contrived example:
$ (printf "%s\t%s\n" 9 B 3 A ; seq 10 | paste - -) | datamash rmdup 1
9 B
3 A
1 2
5 6
7 8
And on the other hand, because 'datamash' is non-standard,
there's less of a problem in adding new functionality (i.e. "bloat" is
not as big as a concern as it is for coreutils).
Hope this helps.
regards,
- assaf
