OK I just update Spout SDK to version 2.006 but this do not fix the pixel format error.
So as a workaround in case of unknown pixel format, I use DXGI_FORMAT_R8G8B8A8_UNORM by default.
(I think SpoutReceiver do the same)
This seems to do the job with Smode using DX11 !
Here the new version:
http://download.smode.fr/synth/beta/Smode%20Synth-master-7.7.26.f6a32ec-win64.exe
I think developers from VVVV should considere to update Spout SDK to version 2.006 too to avoid this kind of issue with other software.
The strange thing is that SpoutPanel considere VVVV enforce DX9, may be it is related...