You are using / at the beginning of your error document path, which means "root", so perhaps you should try add the directory:
ErrorDocument 404 /gallery/error.html
Note, since you want have two different error pages, this line should go into .htaccess in your gallery root directory (/gallery/.htaccess)