Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Programming/Scripts

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 4th December 2005, 03:06
Wire323 Wire323 is offline
Junior Member
 
Join Date: Dec 2005
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Default Bash - Deleting duplicate records

I have a text file full of user-submitted email addresses. I want to remove the duplicate records, but it isn't as simple as using "uniq." When I find a dupe I want to remove both of them, not just one. If it's possible I'd also like to create a text file containing all of the email addresses that had duplicates.

Is this possible?

Thanks
Reply With Quote
Sponsored Links
  #2  
Old 4th December 2005, 03:51
Wire323 Wire323 is offline
Junior Member
 
Join Date: Dec 2005
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Default

I've changed things slightly. Instead of removing them completely I'd like to leave on, and only take the dupes out. I know I can do that with uniq, but how would I know which ones were taken out so I can write them to a file?
Reply With Quote
  #3  
Old 4th December 2005, 05:56
Wire323 Wire323 is offline
Junior Member
 
Join Date: Dec 2005
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Default

I don't know if this was the best way, but I was able to do it like this:

sort participants | uniq > temp1
sort participants > temp2
comm -1 -3 temp1 temp2 > temp3
sort temp3 | uniq > outputfile
Reply With Quote
  #4  
Old 4th December 2005, 11:37
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,732 Times in 2,569 Posts
Default

Quote:
Originally Posted by Wire323
I don't know if this was the best way
If it works it's ok!
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #5  
Old 8th March 2006, 13:26
muha muha is offline
Junior Member
 
Join Date: Mar 2006
Posts: 2
Thanks: 0
Thanked 0 Times in 0 Posts
 
Default

An old post but heh, thought i might add a bit:
To show only unique lines from <file>:
Code:
$ uniq file
To show only the non-unique lines once:
Code:
$ uniq -d file
If the lines are not ordered yet. So remove non-consequtive duplicate lines spread out through the file:
Code:
$ sort file| uniq

Last edited by muha; 8th March 2006 at 13:31.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Look up DNS records xtothez Technical 3 23rd August 2012 15:04
Deleting DNS master records jernis Installation/Configuration 1 22nd September 2005 12:11


All times are GMT +2. The time now is 19:34.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.