Secret key should be passed as Uint8Array. nacl will give you the keys in the correct format already.

You can also do the following:
const secretKey = TonWeb.utils.hexToBytes('YOUR HEX');
const keyPair = TonWeb.utils.nacl.sign.keyPair.fromSecretKey(secretKey);

On the other side, you can check the detail here like in Jetton Example by using Tact Language
https://github.com/ton-community/tact-jetton/blob/21e198cd7c95a287509f91f806f1ed32a4252129/sources/jetton.deploy.ts#L25