Here's my experience with streaming to PS3:
MPEG2 files have no size limit. Audio should be AC3 or LPCM.
M2TS files have no size limit. You can stream anything that's MPG2 or AVC. VC1 doesn't work. Audio: AC3 and LPCM works. TrueHD,DTS doesn't work.
I suggest you convert MKV files to MPEG.
http://www.pavtube.com/guide/convert-mkv-dts-to-ps3.html