> For the complete documentation index, see [llms.txt](https://docs.rasgoml.com/rasgo-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rasgoml.com/rasgo-docs/rasgo-0.1/all-transforms/remove_duplicates.md).

# Remove Duplicates

Deduplicate a table based on a passed-in composite key. Once an order column and an order method are selected, only the top record from the resulting grouped and ordered dataset will be kept.

## Parameters

| Name          | Type            | Description                                                                  | Is Optional |
| ------------- | --------------- | ---------------------------------------------------------------------------- | ----------- |
| natural\_key  | column\_list    | Columns forming the grain at which to remove duplicates                      |             |
| order\_col    | column\_list    | Columns by which to order the result set, such that the first result is kept |             |
| order\_method | sort\_direction | Sets the order behavior for the chosen `order_col`. Can be ASC or DESC.      |             |

## Example

```python
ds = rasgo.get.dataset(id)

ds2 = ds.remove_duplicates(
  natural_key=["FIPS", "DS_WEATHER_ICON", "DATE"],
  order_col=["DATE", "FIPS"],
  order_method="asc"
)
ds2.preview()

```

## Source Code

{% embed url="<https://github.com/rasgointelligence/RasgoTransforms/blob/main/rasgotransforms/rasgotransforms/transforms/remove_duplicates/remove_duplicates.sql>" %}