{"id":14307,"date":"2014-10-22T00:47:35","date_gmt":"2014-10-22T05:47:35","guid":{"rendered":"http:\/\/bucktownbell.com\/?p=14307"},"modified":"2014-10-22T00:47:35","modified_gmt":"2014-10-22T05:47:35","slug":"remove-duplicates-without-sorting-file","status":"publish","type":"post","link":"http:\/\/bucktownbell.com\/?p=14307","title":{"rendered":"Remove duplicates without sorting file"},"content":{"rendered":"<blockquote><p>Usually whenever we have to remove duplicate entries from a file, we do a sort of the entries and then eliminate the duplicates using &#8220;uniq&#8221; command.<\/p>\n<p>But if we have to remove the duplicates and preserve the same order of occurrence of the entries, here is the way:<\/p><\/blockquote>\n<p>via <a href=\"http:\/\/www.unixcl.com\/2008\/03\/remove-duplicates-without-sorting-file.html\">UNIX Command Line: Remove duplicates without sorting file &#8211; BASH<\/a>.<\/p>\n<blockquote><p><span style=\"background: silver none repeat scroll 0% 50%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;\">$ awk &#8216; !x[$0]++&#8217; file3<\/span><\/p><\/blockquote>\n<p>From: <a class=\"question-hyperlink\" href=\"http:\/\/stackoverflow.com\/questions\/11532157\/unix-removing-duplicate-lines-without-sorting\">Unix: removing duplicate lines without sorting<\/a><\/p>\n<blockquote><p>This command is simply telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array named x is incremented and the line printed if the content of that node was not (!) previously set.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>Usually whenever we have to remove duplicate entries from a file, we do a sort of the entries and then eliminate the duplicates using &#8220;uniq&#8221; command. But if we have to remove the duplicates and preserve the same order of &hellip; <a href=\"http:\/\/bucktownbell.com\/?p=14307\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35],"tags":[273,1093,28,283],"class_list":["post-14307","post","type-post","status-publish","format-standard","hentry","category-operating-systems","tag-awk","tag-bash","tag-linux-command","tag-tech-blog"],"_links":{"self":[{"href":"http:\/\/bucktownbell.com\/index.php?rest_route=\/wp\/v2\/posts\/14307","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/bucktownbell.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/bucktownbell.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/bucktownbell.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/bucktownbell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14307"}],"version-history":[{"count":1,"href":"http:\/\/bucktownbell.com\/index.php?rest_route=\/wp\/v2\/posts\/14307\/revisions"}],"predecessor-version":[{"id":14308,"href":"http:\/\/bucktownbell.com\/index.php?rest_route=\/wp\/v2\/posts\/14307\/revisions\/14308"}],"wp:attachment":[{"href":"http:\/\/bucktownbell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14307"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/bucktownbell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14307"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/bucktownbell.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14307"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}