FireWall between weblogic & DataBase server may cause weblogic hang!!!

some time we meet weblogic hang ,we use thread dump to detect the reason:

3XMTHREADINFO "ExecuteThread: '10' for queue: 'default'" (TID:0x302C39C8, sys_thread_t:0x3F6AB8B0, state:R, native ID:0x1011) prio=5
4XESTACKTRACE at java.net.SocketInputStream.socketRead(Native Method)
4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
4XESTACKTRACE at oracle.net.ns.Packet.receive(Unknown Source)(Compiled Code)
4XESTACKTRACE at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)(Compiled Code)
4XESTACKTRACE at oracle.net.ns.NetInputStream.read(Unknown Source)(Compiled Code)
4XESTACKTRACE at oracle.net.ns.NetInputStream.read(Unknown Source)(Inlined Compiled Code)
4XESTACKTRACE at oracle.net.ns.NetInputStream.read(Unknown Source)(Compiled Code)
4XESTACKTRACE at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java(Compiled Code))
4XESTACKTRACE at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java(Inlined Compiled Code))
4XESTACKTRACE at oracle.jdbc.ttc7.Oopen.receive(Oopen.java(Compiled Code))
4XESTACKTRACE at oracle.jdbc.ttc7.TTC7Protocol.open(TTC7Protocol.java(Compiled Code))
4XESTACKTRACE at oracle.jdbc.driver.OracleStatement.(OracleStatement.java(Compiled Code))
4XESTACKTRACE at oracle.jdbc.driver.OracleConnection.privateCreateStatement(OracleConnection.java(Inlined Compiled Code))
4XESTACKTRACE at oracle.jdbc.driver.OracleConnection.createStatement(OracleConnection.java(Compiled Code))
4XESTACKTRACE at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java(Compiled Code))
4XESTACKTRACE at weblogic.jdbc.common.internal.ResourceAllocator.reserve(ResourceAllocator.java(Compiled Code))
4XESTACKTRACE at weblogic.jdbc.common.internal.ResourceAllocator.reserve(ResourceAllocator.java:658)
4XESTACKTRACE at weblogic.jdbc.common.internal.ResourceAllocator.reserveNoWait(ResourceAllocator.java:616)
4XESTACKTRACE at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:165)
4XESTACKTRACE at weblogic.jdbc.common.internal.ConnectionPool.reserveNoWait(ConnectionPool.java:127)
4XESTACKTRACE at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:259)
4XESTACKTRACE at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:292)
........


we also found many other users have this same problem,it seems oracle connection hang ,later , we find the firewall will clear the unuse connection ,but the wls pool (or JDBC driver)will not detect it,so the hang occur

in weblogic 7.0 ,in configuration of pool ,we can set up refresh period, it will help us to use those unused connection in the period

deadlock on create new jdbc connection from weblogic conn pool

On weblogic6.1


FOUND A JAVA LEVEL DEADLOCK:
----------------------------
"ExecuteThread: '10' for queue: 'default'": ==========
at
weblogic.common.internal.ResourceAllocator.makeResources
...
at weblogic.jdbc.pool.Driver.connect(Driver.java:152)
at java.sql.DriverManager.getConnection(DriverManager.java:512)

lock with

"ExecuteThread: '22' for queue: 'default'": ==========
at java.sql.DriverManager.println(DriverManager.java:419)





for this problem ,I quote a anwser from weblogic.developer.interest.jdbc newsgroup:


DriverManager.getConnection() is a class-synchronized
method, if your code calls it, and then our pool needs to also
make that call, it can lead to deadlocks, and also, any other JDBC
that may be running in the server already ahving a connection may
also halt, because drivers use DriverManager.println() to log their
actions and exceptions, and that method is also class-synchronized.
(Send complaints to JavaSoft).
We try to avoid calling DriverManager.println() in our pool code,
calling Driver.connect() directly, but for some drivers, this may
fail, so we fall back on DriverManager.getConnection().
All in all, it is much better for performance and stability to have all your
pool connecitons pre-made.

Driver d = (Driver)Class.forName("weblogic.jdbc.pool.Driver").newInstance();
Connection c = d.connect("jdbc:weblogic:pool:myPoolName", null);


Joseph Weinstein

Note the errors :when put firewall between the proxy and cluster On weblogic cluster architecture

when we put a firewall (who do the IP translation) between the proxy and cluster(a few managered server),we will meet this error :


The client initiates contact with the WebLogic Server cluster by requesting a connection to the first server at 205.20.xxx.100:7001. The firewall translates this address and connects the client to the internal IP address of 205.20.yyy.100:7001.

The client performs a JNDI lookup of a pinned Object C that resides on the third WebLogic Server instance in the cluster. The stub for Object C contains the internal IP address of the server hosting the object, 205.20.yyy.300:7001.

When the client attempts to instantiate Object C, it requests a connection to the server hosting the object using IP address 205.20.yyy.300:7001. The firewall denies this connection, because the client has requested a restricted, internal IP address, rather than the publicly-available address of the server.

the solution is setting the externalDNSName .See Server—>Configuration—>General in Administration Console

ref link :
http://edocs.bea.com/wls/docs70/cluster/planning.html

in this page , see the Avoiding Problems ->Firewall Considerations

Preventing and Handling Duplicate Messages ON onMessage() method

Incidentally ,we found our MDB handling duplicate messages ,why does it happen?

from WLS online doc,I get the infos:

"If the MDB receives the message, but fails to send an acknowledgement, the JMS producer re-sends the same message ,even the onMessage() method successful completed

So ,If the MDB manages its own transactions, then your onMessage() code must be prepared to handle duplicate messages"

The resolve way is Using the CMT on MDB
1. Set the transaction-type element inside the message-driven element in the ejb-jar.xml file to Container.

2. Set the trans-attribute element inside the container-transaction attribute in the ejb-jar.xml file to Required.

This means message receipt and acknowledgement happen within a transaction; either both happen or neither happens.


Ok , it 's very simple, But in this way ,what's difference between the synchronous & asynchronous in this scenario, asynchronous in a big transaction is looks like a synchronous method with local trans, decouple a little?

IMHO,Customer will take a good think in this scenario

*******************************************************
another notes

when using the MDB ,you should take care the sender 's transaction setting ,when set the jmsConnectionFactory,Don't forget the UserTransactionsEnabled option,which means whether send method is JTA aware

jspc error when meet the struts define tag ON Weblogic8.1 sp1

Customer report , they run the Struts "define tag" on Weblogic8.1 sp1 , but the weblogic omitted <bean:define....>operation

but the same thing on WLS 7.0 is OK

"
we replay the error when we run the Struts example struts-exercise-taglib.war

use the <bean:define> tag
"


when weblogic8.1 jspc(now is APPC) compile any jsp which use the tag , the error occur

now , I have not any idea for it , just wait BEA CS response


==========resolve info======================

it 's seems WLS version control error: we get the rolling patch very soon,it contains the new tag support class

anyone who meet the same error , can contact me for the patch

I belive BEA will renew it in WLS next service patch or else
==========

Server VM & NTSocketMuxer caused deadlock on win2000 sp3 when running WLS

our customer report WLS hang on win2000:(thread dump) :



JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:
------------------------------------------------
Java Stack for "ExecuteThread: '13' for queue: 'default'":
==========
at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:572)
- waiting to lock <44f75d8> (a weblogic.socket.IORecord)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:
23)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)
Java Stack for "ExecuteThread: '9' for queue: 'default'":
==========
at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:572)
- waiting to lock <2fd5a78> (a weblogic.socket.IORecord)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:
23)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)

Found 1 deadlock.



=======resolve info ==============


we look for the bug at SUN bug list :

seems it 's Bug: 4854464

http://developer.java.sun.com/developer/bugParade/bugs/4854464.html


"JDK 1.3.1_06 hangs/deadlocks on Windows2000 SP3 server. This problem
is specific to server VM, and does not happen using -classic option.
Problem happens using jdk 1.3.1_02 through 1.3.1_06 (not yet sure
about jdk 1.3.1_08), and also happens with jdk 1.4.1_01 (primarily in
Weblogic environment).
"

but the bug status is still In progress, so it give 2 method to overcome it :

1) Use -classic option instead

2) Excluding a particular method using .hotspot_compiler:
exclude weblogic/socket/NTSocketMuxer processSockets


=================================

Use the servlet filter to protect static& dynamic web contents

Use the servlet filter to protect static& dynamic web contents

My customer want to using web.xml configuration to protect static& dynamic web contents

Wls support the login-config to do this , but this depend the j_security_check,can only authen inner role & principle

if you want to use external authen , we advise using the servlet filter to so this :


in web.xml :



<filter>
<filter-name>loginFilter</filter-name>
<filter-class>loginFilter</filter-class>
</filter>

<filter-mapping>
<filter-name>loginFilter</filter-name>
<url-pattern>/admin/*</url-pattern>
</filter-mapping>


loginFilter.java 's doFilter method :

public void doFilter(ServletRequest req, ServletResponse res, FilterChain fc)
throws java.io.IOException, javax.servlet.ServletException {

HttpSession sess = ((HttpServletRequest)req).getSession(true);
String auth = (String)sess.getAttribute("authenticated");

if(auth == null || auth.equals("false")) {

((HttpServletResponse)res).sendRedirect("/login.jsp?path_context="+URLEncoder.encode(((HttpServletRequest)req).getRequestURI()+"?"+((HttpServletRequest)req).getQueryString()));
return;
}


fc.doFilter(req,res);
}



then wrapper the authen in the authenticate.jsp ,which is the login.jsp 's action file


<%@page import="java.net.URLDecoder"%>

<%
//authen the user
//.....put your authen method , I just give a example

boolean authenticated=false;

if (request.getParameter("username").equals("abc") && request.getParameter("password").equals("abc") )
authenticated = true;

//*******************
if (authenticated) {
(request.getSession(false)).setAttribute("authenticated", "true");
response.sendRedirect(URLDecoder.decode(request.getParameter("path_context")));
}
else {
(request.getSession(false)).setAttribute("authenticated", "false");
response.sendRedirect("/error.jsp?path_context="+request.getParameter("path_context"));
}

%>


finally the login.jsp :

<form method="post" action="/authenticate.jsp">
<table border="0" bgcolor=#eeeeee align=center cellspacing=10>
<tr>
<td>Username:</td>
<td>
<input type="TEXT" name="username">
</td>
</tr>
<tr>
<td>Password:</td>
<td>
<input type="PASSWORD" name="password">
</td>
</tr>
</table>
<p>
<center>
<input type="SUBMIT" name="submit" value="Login">
<input type="hidden" name="path_context" value="<%=request.getParameter("path_context")%>">
</center>
</form>