Sell custom entity via custom line item

Hello,

I find myself in a vicious circle trying to sell custom entity via custom line item.
Not sure if this was covered already somewhere.
The goal here is to avoid adding the custom entity via Commerce Product because it will bring another layer to the already existing&complicated data structure.

I have a custom line item that has reference to my custom "for sale" entity.

1. If the custom line item is not product type: The problem is that you can't checkout if you do not have at least one product type line item in your order.

2. If the custom line item is a product type: Since I don't actually have a reference to a product(field is empty), the system will fail when trying to calculate the prices throwing some SQL Exception. If I would populate the commerce product reference field this will mess up / overwrite my custom prices.

I am not sure what is the best way to be able to have custom line items, referencing my own entities.
I ended up doing ugly things like having a fake-product present in all orders that is always hidden... :)

Any insight?
Thanks,
Mihai