We don't want users to directly connect to our application's database using ODBC or Sybase tools. When creating user logins for our application the specified password will be modified by an internal algorithm before the corresponding database user is created. The password the user enters when logging in to our applictaion is treated the same way before using the credentials to connect to the database.

So the combination of login name and password will not be valid for connections made outside our application.

You could use a login procedure in combination with the APPINFO connection property to reject connections with inappropriate APPINFO values.

You could thereby require that the EXE value is restricted to your EXE name, or exclude particular tools like Sybase Central, or could add an APPINFO=MySecret connection parameter to the connection string.

(Note: APPINFO is both the connection_property including several entries like API=, EXE=, and a connection parameter that builds the APPINFO= entry in the connection_property. Somewhat confusing, methinks.)

But note that won't prevent advanced attacks: In case you use a "magic" APPINFO that will be stored somewhere and could by used by Joe Hacker, too, and if you restrict to a particular EXE name, well, I guess one could rename Sybase Central to that, too...

A very simple (and untested) sample based on the login procedure sample from the docs - it should prevent any client connection unless APPINFO=MySecret is specified in the connection string (or the exe's path contains that string...):

CREATE PROCEDURE DBA.login_check( )
   BEGIN
      DECLARE INVALID_LOGON EXCEPTION FOR SQLSTATE '28000';
      DECLARE MagicValue VARCHAR(100) = 'MySecret';
      IF (CONNECTION_PROPERTY('AppInfo') NOT LIKE '%' || MagicValue || '%') THEN
          SIGNAL INVALID_LOGON;
      ELSE
          CALL sp_login_environment;
      END IF;
   END
go

GRANT EXECUTE ON DBA.login_check TO PUBLIC
go

SET OPTION PUBLIC.login_procedure='DBA.login_check'
go