You just want to remove duplicates ?!
Code:
sort file.txt | uniq
''LC_ALL=C sort -u''
You can also ''man sort'' and look at the --parallel and -S switches to improve performance.
ah, i see, its a unix command.
you had me thinking you were a wizard there for a second.
This was driving me nuts. I came back to this post later and installed cygwin and finally got it to work. Funny how sometimes you get a wild hair up your ass about something and can't let it go.
You can also use WSL and run Ubuntu under Windows 10