A boc is not limited in size, in fact the whole blockchain state is a cell with a bunch of cells wrapped within them. Everything is a cell!
An external message is limited by 64kb, which may limit the size of a boc that you're trying to create with your smart contract logic. You can see this defined in the lite client, but it is subject to change:
