Only raise DeferredImportError if we are not building docs

Only raise DeferredImportError if we are not building docs #54