Custom DNS is designed primarily to work with domains owned/registered by the customer (example.com, example.co.uk, etc). However, Custom DNS can also be used with a subdomain of such a domain (host.example.com, host.example.co.uk, etc).
If you have, for example, registered the domain example.com and you want Custom DNS to provide DNS for only a portion of this domain, you can delegate a host or subdomain within this domain to Custom DNS by creating the necessary NS records.
For example, if you wanted Custom DNS to handle the subdomain dynamic.example.com, in the DNS for example.com you would create the following entries:
dynamic.example.com. 48600 IN NS ns1.mydyndns.org. dynamic.example.com. 48600 IN NS ns2.mydyndns.org. dynamic.example.com. 48600 IN NS ns3.mydyndns.org. dynamic.example.com. 48600 IN NS ns4.mydyndns.org. dynamic.example.com. 48600 IN NS ns5.mydyndns.org.
Note the trailing "." after the hostname, as these are important (vital).