I don't think their disliked because of damage output, I think most people dislike them because they just don't have variety. You really have 1 option with the vanilla swashbuckler, and none of the archetypes open up many options either. If you're a swashbuckler, you're pretty much locked into using a rapier, a cutlass, a kukri, or a scimitar and you will go crit fishing. Anything else is going to do a really bad job.

That's not to say it's a bad class as a whole, it just doesn't have options. You will pick up your 18-20 crit weapon, you'll give it keen until you can get Crit Focus, and you'll give it Dex to damage, because that's what the class was made to do. On the other hand, a fighter has the ability to pick up different combat maneuver training, he can choose a bunch of different weapons he can use, he can choose to fight with 1 hand and a shield, 2 weapons, or with a 2-handed weapon. A gunslinger can choose to fight with pistols at close range, muskets at long range, or spread the love with a blunderbuss, he can get up close with a martial weapon if need be, he can use his gun for other purposes. A ranger can fight with 2 core styles and 8 other styles from the APG and Ultimate Intrigue, can choose to fight with their animal companion, can focus on certain terrains and enemies. Paladins have plenty of support options and can focus on any fighting style they want because the bonuses they get from class abilities can supplement them until they have the feats to finish it. I could go on, but you get the gist.

Essentially, Swashbucklers are "low tier" simply because they have less variety. They have one thing, and they do it incredibly well, but they are predictable. Casters are high tier because they can adapt to any situation, fighters are higher than swashbucklers because fighters gain more options in their martial combat, and swashbucklers are low tier because negating crits and enemies who like to use maneuvers shut them down hard.

Also, because you can't get Dex to damage on a Sword Cane RAW. I wanted to have a pimp cane but it's not viable!