If you see a sudden growth in msdb after working with SQL Broker it probably means, that the Service is sending message to the Queue and not able to succeed.
sysxmitqueue object will be occupying the space. To clear that, use
ALTER DATABASE [msdb] SET NEW_BROKER WITH ROLLBACK IMMEDIATE;
After that Shrink the dabatase if you need to release the space.