Usually whenever we have to remove duplicate entries from a file, we do a sort of the entries and then eliminate the duplicates using “uniq” command.

But if we have to remove the duplicates and preserve the same order of occurrence of the entries, here is the way:

via UNIX Command Line: Remove duplicates without sorting file – BASH.

$ awk ‘ !x[$0]++’ file3

From: Unix: removing duplicate lines without sorting

This command is simply telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array named x is incremented and the line printed if the content of that node was not (!) previously set.