Yes, as you said: Remove {download_button} and {download_zip_button} fro the templates and set the Permission for Download to "Administrators".
If you want that this is preselected if you add new categories, change in admin/categories.php
"auth_download" => AUTH_USER,
to
"auth_download" => AUTH_ADMIN,
Greets Jan