Kyplex

I have read a great article about sorting command tricks and would like to share one of the tricks with you.

Let's say you have non-unique, textual file from which you need to remove duplicate strings preserving file order. One will suggest to do it in Perl, though it could be done using basic shell commands:

cat file.txt | nl | sort -k2 -u | sort -n | cut -f2 > result.txt

Run the above shell command as follows:

1. Read the file.txt file.
2. Line numbers are added to each line of the original file.
3. Sort and remove duplicate strings using the second field as if it were the original file.
4. Resort the results according to the line number.
5. Remove the line numbers from the results.
6. Save the results.

Note
The nl command adds line numbers to the text file.

Bookmark/Search this post with:

 •    •    •    •    •    •    •    •    •    •