So many programmers make this mistake because they don’t seem to get it. It’s not your fault as a programmer / developer. It’s just that you’re not a DBA.
You need to implement Connection pooling. It makes such a huge difference in the way your DB performance and app works it’s amazing.
So here are some links to connection pooling on a few different Databases:
Mysql: http://dev.mysql.com/tech-resources/articles/connection_pooling_with_connectorj.html
Oracle Connection Pooling: http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
MS-SQL Server: http://msdn.microsoft.com/en-us/library/8xx3tyca.aspx
Postgres: http://www.postgresql.org/docs/7.4/static/jdbc-datasource.htm
PHP: http://php.net/manual/en/mysqlnd-ms.pooling.php
ASP.net: http://msdn.microsoft.com/en-us/library/8xx3tyca%28VS.71%29.aspx
Java: http://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html
Ruby: http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html
Python (Mysql): https://launchpad.net/pysqlpool/
Python (Postgres): http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL
Implement It properly and watch the difference in resource utilization. Especially for high traffic / DB-backed systems.