The answer will likely depend on the implementation, as the Redfish Service may not even get called if the web service/stack handles the certificate/authentication.
But regardless I think you're better off responding with an error if the certificate is invalid (even for resources that don't require authentication), so that the user gets a consistent failure. Otherwise, they may not understand or know the distinction about the Service Root - and will get confused as to why it "works here but not there"...
You are right and Redfish Service may not even get called if the web service/stack handles the certificate/authentication. So redfish service can't give response back and web service/stack itself return error details.
If request comes without authentication for service root and "/redfish" documents, response is provided as per specification.
But if user is giving wrong credentials for basic authentication, Session based as invalid for service root and "/redfish" documents, do we need to return unauthorized or provide the response as per specification.
We discussed that question and didn't think we could get a consistent answer for all implementations because of the different behaviors of the various software stacks. So either response is allowed, but my preference would be to reject the request - because that would provide a more consistent error behavior for the user.