About 3 years ago, I had reported a possible remote DoS attack in Oracle. Oracle addressed the issue and acknowledged me for the discovery in its Critical Patch Update CPU April 2009 . Specifically it is the CVE-2009-0991 entry in the Oracle Database Risk Matrix. We never created any formal public advisory for this, but I was looking for something to post on my blog and writing about it seemed like a good idea :)
There is some flaw in the way Oracle handles user authentication. The authentication process seems to involve an initial TNS CONNECT from the client to which the server responds with an ACCEPT assuming correct server IP and port (1521 is the default LISTENER port) and SID/service name is specified. After which is negotiation of security services and exchange of a couple more packets. At one point, the client seems to send a login packet with username to which the server responds with a key (AUTH_SSKEY) indicating that some sort of a session initiation has begun. This is done whether the username specified in the packet is valid or not. If the client then abruptly terminates the connection as soon as the session key packet is received, the database seems to still keep track of that session internally. On Oracle 10g, sending such multiple login packets with invalid/arbitrary usernames and abruptly terminating connections after the AUTH_SSKEY is received, will very soon exhaust the 'maximum concurrent allowed sessions' limit and thus cause a DoS for other users. On Oracle 9i R2, I could get same results, but the login packet needs a valid username, because in 9i R2 the session key is only generated for valid users. Nevertheless, one can always use known usernames like 'sys', 'dbsnmp' etc
Oracle later confirmed this to be an issue in 11gR1 as well. To resolve the issue, Oracle enhanced the INBOUND_CONNECT_TIMEOUT configuration parameter in CPU April 2009. This parameter allows one to specify a timeout value in seconds to wait until the initiated login process is completed. So, using a small enough value for this parameter, an attacker will not be able to create too many such half-open logon sessions in order to cause the DoS. The default timeout is 60 sec, but it should be less IMHO, although it would really depend on many factors specific to each setup.
New error codes ORA-3135 and ORA-3136 were also added to indicate when the timeout is exceeded, in addition to the existing codes.
Rights to this research belong to my company Application Security, Inc.