As well as hurting business, the expensive wages are costing young people their jobs, he says.
"This is killing small businesses, it's killing employment," Senator Xenophon told reporters in Canberra on Tuesday.
Casual workers, especially students, are missing out on work and hundreds of small businesses around the country are shutting up shop on weekends.
"It's actually a huge disincentive to employ people with penalty rates the way they are," he said.
Under Senator Xenophon's proposed amendment to existing laws, penalty rates would still be payable, but only where an employee has worked more than 38 hours in seven days, or more than 10 hours in a day.
The changes would only apply to small businesses employing fewer than 20 full-time workers.
John Hart, CEO of Restaurant and Catering Australia, said a growing number of businesses were shutting their doors at least one day more than last year, with almost one in five winding back on staff numbers.
"That's bad for everybody," he said.
"It's bad for consumers because they can't dine at their venues of choice, it's bad for the staff because the staff aren't getting the hours they need, and it's bad for our business operators who are making less money.
"The reality is unless we stem penalty rates on weekends, we're going to have an industry that is shutting down progressively."
Hart said if weekend pay rates were closer to weekday rates, restaurants would likely drop their Sunday and public holiday surcharges.
Peter Strong from the Council of Small Business - who recently shut his own bookshop on Sundays - says casual workers are happy to work for less and it's a mistake to legislate double-time penalties.
"We've got to look our staff in the eye and say `I'm sorry, but you don't have a job any more ... because the unions or the government has said that you should get double time, which you know I can't afford'," he said.
"'I know you're quite happy with time-and-a-half, but somebody has said you shouldn't be'."
