XFCE stores it's configuration for the running session in xfconfd. Feel free to back up the files you're going to delete first.

    Shut down the panel first, xfce4-panel --quit
    Kill the xfce4 configuration daemon, pkill xfconfd
    First delete settings for the panel, rm -rf ~/.config/xfce4/panel
    Clear out the settings for xfconfd, rm -rf ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml
    Restart the panel, run xfce4-panel. This will respawn xfconfd automatically. Note if you need or want to restart xfconfd manually know that on my installation it was in /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd which was outside of $PATH.

This clears it for the running session, regenerates the files, and sets up the default for future sessions.
Want it in one line?

xfce4-panel --quit ; pkill xfconfd ; rm -rf ~/.config/xfce4/panel ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml ; xfce4-panel;