trusted_certificates is a Long VarChar and should contain 'file=' the full path (relative to the Sybase server) and the file name, otherwise secure=1 will default to the OS certificate store. I also believe that you can simply use the file path as the trusted_certificates single parameter.

Mail servers are now getting very picky about authentication. Most will accept without a certificate, but will insist upon encryption, and will look for TLS 1.2 or TLS 1.3 as the standard. If the test one from your PC works, it is probably through an ISP that has not caught up to the latest standards, or it found a valid certificate in your store and used a later encryption method. If you try connecting to a mail server like o365 it will be rejected if your customer has outdated SSL and/or TLS. But the error indicates that it could not find a default file to open.

When you send a test email check the headers to see the encryption type - you should see something like this...

(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 256/256 bits) (Client did not present a certificate) by CMGW with ESMTP

Also, bear in mind that now mail will either not be accepted, or end up in spam, if DMARC, SPF and DKIM are not set.

Additional edit

For what it's worth, I've sent email using other SMTP APIs, and it's pretty easy. Nothing to do with 'trusted certificates'.

To clarify - trustedcertificates has an option not use one - you don't generally need a certificate to connect to an ISP mail server, but if you want a secure connection then the encryption method must be a later type. Specifying trusted_certificate=none will make a secure TLS connection without rendering a certificate.

As another edit

Simply leaving out trusted_certificates, or setting it to null, with 'secure=1' it will connect with TLS. I just tried it with only these parameters.

BEGIN
DECLARE return_Code INTEGER ;
return_code = CALL xp_startsmtp(
smtp_sender = 'You@YourDomain.com',
smtp_server = 'smtp.ionos.com',
smtp_port = '587',
smtp_sender_name = 'My Name',
smtp_auth_username = 'MyUserName',
smtp_auth_password = 'MySecretPassword',
secure = 1
);

@return_code = CALL xp_sendmail ( 
recipient     = 'Test@SomeOtherDomain.com',  
subject       = 'Test Subject',  
"message"     = 'Test message' );

CALL xp_stopsmtp();

select STRING('smtp return code = ', return_code, ' ', 'Error Code = ',  xp_get_mail_error_code(), ' ', 'Return Text = "', xp_get_mail_error_text(), '"') as 'SMTP returned values'

END