Duplicate removal in Excel
Today’s IT landscapes are characterized by heterogeneous databases, quite a challenge for managing data quality. Quality checks must thus be a routine. One important method: Duplicate removal. Here’s how to get duplicate removal right in Excel.
Often you run into a situation where you want to identify the unique records in a worksheet and maybe delete the duplicates.
Method 1: Use Remove Duplicates icon (works only in Excel 2007 and above)
This method is irreversible! Please make a copy of your data first to another blank section of the worksheet.
1.Select any cell in your data set.2.Go to Data ->Remove Duplicates.3.From the Remove Duplicates dialog choose the columns which should be considered. For instance, if records where both the Sales Rep and Region were identical are to be removed, check the box for both fields. If you are only trying to get a unique list of Sales Reps, choose only the Sales Rep field.
It will report that n1 duplicates were removed and n2 records remain.
Method 2: Use Advanced Filter with the Unique Option
1.To the right of your data, in cell F1, copy the heading from the Sales Rep column where you want to find unique values.2.Select a cell in your data set.3.In Excel 97-2003,Go to Data ->Filter ->Advanced Filter. In Excel 2007 and above, go to Data ->Sort & Filter group ->Advancedicon.4.This will bring up the Advanced Filter dialog box.
5.You can choose one of two radio button options. Either Filter the list, in-place or Copy to another location.6.If you choose to Filter the list in-place, with the Unique records only checkbox checked,all duplicate records will be lost.7.If you choose Copy to another location, thenIn the Copy To box, specify the copy of your heading. In the Figure, this is cell F1.8.Click the box for Unique Records Only and click OK.
Excel will provide you a unique list of customers in column F.
Method 3: Use a Formula to Determine if a Record is Unique
The COUNTIF function can count how many records above the current record match the current record. To make this work, you need to use absolute addressing using the ‘$’ symbol. Copy this formula to the remaining cells. The formula in E9 will read: =COUNTIF(C$1:C8,C9)=0.
You can now sort column E and the unique values will be at the top or bottom of the list depending on the sort order.
Method 4: Use a Pivot Table to get Unique Customers
A pivot table is great at finding unique values. This is the fastest way.
1.Select a cell in your data set.2.In Excel 2000-2003, go to Data ->Pivot Table and Pivot Chart Report and click on Finish. In the Pivot Table Field List, click on the Customer Field. Click the Add To button.3.In Excel 2010 go to Insert ->Pivot Table-> Pivot Table. Select Region followed by Sales Rep as the Row Labels of the Pivot Table.
Excel will show you a unique list of Sales Reps by Region.
In many cases you may have specific requirements for duplicate removal, such as giving a list of removed duplicates in a separate list, or you may want to introduce certain exceptions or rather than removing duplicates you would want them only highlighted with a certain background colour. You may realize that a formula-based approach for duplicate removal applied to huge amounts of data won’t work efficiently. In these cases, a macro-based method can be implemented quickly and easily. Please check our microtools forExcel on our website www.solutiontogo.de. We like to call them microtools because they can be easily added to any application … and development takes only a few hours!