And I am the exact opposite to BHT. My homemade handles are sanded to about 120 grit and then left alone. In time they will, as already mentioned on this thread, take on a patina just from handling. This finish may not look the prettiest but the 'Feel' is second to none.
My advice...… Don't remove the handles unless you are going to make new ones. Decide your own priorities (BHT's or mine) and go for it, there isn't a right or wrong.
Some years ago, incidentally, I mounted a tool between centres by driving the cutting edge into a softish scrap of wood in the headstock. It took a bit of faffing to get the handle to run true(ish) but I did manage to joint an extension onto the existing, too short, handle. Might have been easier to make a replacement but hey, the idea struck so I had to give it a go.
All the Best
Brian