Why RPC 116 error and/or Authentication failure errors on CS 5.3 SP4 and SP5?

Symptoms

5.3 SP4 and SP5 Clients will see "RPC 116 error" and/or "Authentication failure".

This error is only seen with trace -otrace_authentication and -oticket_trace options:
Start-AuthenticateUserByTicket:UserLoginName(testuser1),
TICKET TRACE: dmLoginTicketMgr::VerifyTicket() : encodedBuffer = DM_TICKET=AAAAAgAAAOQAAAAKAAAAFUeUZYRHlGawAAAAOGxzY21zAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEZyYW4gU2Nod2lldHprZQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGxzY21zAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGx2Y21zMDEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGNlaWxpbmcxMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAM0pyOTdRTDJOakFUbVhwak15Y21CWDdLbG5zYnM2aUVUU1pXUy9kbHdwWjlzcllmaGIrNFZnPT0=
TICKET TRACE: dmLoginTicketMgr::LoadTicket() : Login ticket successfully loaded into dmLoginTicket struct.
TICKET TRACE: Failed to verify login ticket because user name mismatch: ticket.m_userName=tsestuser2, userName=testuser1
End-AuthenticateUserByTicket:
failure

Cause

If a ticketed session is timed out, the generated ticket would be different from the user who actually uses the ticket to establish the connection. Also setting the wrong password in the server's session causes subsequent server reconnects to fail in the case of client session timeout.

Resolution

This issue is resolved in CS 5.3 SP6, otherwise an eng patch request must be submitted.

Determining if you are having this issue:
Perform the following 2 tests with tracing turned on.

# To enable ticket trace
API> apply,c,NULL,SET_OPTIONS,OPTION,S,ticket_trace,VALUE,B,T

# To disable ticket trace, after the test are completed.
API> apply,c,NULL,SET_OPTIONS,OPTION,S,ticket_trace,VALUE,B,F
Send us the content server log file with ticket trace info.

DESCRIPTION OF UNIT TESTING TO BE TO VERIFY CHANGE:

This is an example, do not use the ticket generated in this example for your test.
Just use the api commands listed.

Test 1:
=====
Run IAPI to connect to docbase as super user.
Make sure the login ticket timeout value is set to 5 minutes in serverconfig object.
Make sure the DMCL session timeout value is also set to 5 minutes.

Generate a login ticket for any non-super user, say tuser1 (make sure tuser1's password is different from that of super user).

API> getlogin,c,tuser1
...
DM_TICKET=AAAAAgAAAOQAAAABAAAAAUcxydlHMcsFAAAAOHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAHR1c2VyMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHJycHZtaW5kZXgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAFJSUFZNSU5ERVgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAG1hbmFnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaUkzL25xYXZPZzdMWEY4WWVNajg0bVpNQVpyZ3BJT1JGcXZIY0w0V1h1S1oyMDRsY2x2ZnNnPT0=

Dump login ticket to see expiration date.

API>dumploginticket,c,DM_TICKET=AAAAAgAAAOQAAAABAAAAAUcxydlHMcsFAAAAOHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAHR1c2VyMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHJycHZtaW5kZXgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAFJSUFZNSU5ERVgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAG1hbmFnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaUkzL25xYXZPZzdMWEY4WWVNajg0bVpNQVpyZ3BJT1JGcXZIY0w0V1h1S1oyMDRsY2x2ZnNnPT0=
...
LOGIN TICKET DUMP
==========================================
Version : 5.3 (ticket version 2)
Scope : global
Sequence Number : 0000000001
Single Use : No
Create Time : Wed Nov 07 06:21:13 2007
Expiration Time : Wed Nov 07 06:26:13 2007
User : tuser1
Password : *********
Domain : rrpvmindex01
Server : vmcs01_535_ora10g203
Docbase : vmcs01_535_ora10g203
Host : RRPVMINDEX01
Connect to docbase as "tuser1" using the newly generated ticket.

API>connect,vmcs01_535_ora10g203,tuser1,DM_TICKET=AAAAAgAAAOQAAAABAAAAAUcxydlHMcsFAAAAOHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAHR1c2VyMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHJycHZtaW5kZXgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAFJSUFZNSU5ERVgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAG1hbmFnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaUkzL25xYXZPZzdMWEY4WWVNajg0bVpNQVpyZ3BJT1JGcXZIY0w0V1h1S1oyMDRsY2x2ZnNnPT0=
...
s1

********************************************************
IMPORTANT:
Wait for at least 10 minutes to make sure the ticket is expired, and the DMCL client's connection to docbase/content server is timed out ********************************************************

Then, try to re-connect to content server after client session is timed out and ticket is expired, by trying to create a dm_document object.

API> create,s1,dm_document
...

Make sure you do not get the following error message, you should get a new object_id after the "create" command is issued.

[DM_API_E_NOTYPE]error: "Type name 'dm_document' is not a valid type."
[DM_SESSION_E_START_FAIL]error: "Server did not start session. Please see your system administrator or check the server log.
Error message from server was:
[DM_SESSION_E_RPC_ERROR]error: "RPC error 116 occurred: Unknown error code 116 (_nl_error_ = 0). Extended network error: 0"
[DM_SESSION_E_AUTH_FAIL]error: "Authentication failed for user tuser1 with docbase vmcs01_535_ora10g203.""


Test 2:
=====
Run IAPI as super user to create "TestMethod" dm_method object.
-------------------------------------------------------
create,s0,dm_method
set,s0,l,object_name
TestMethod
set,s0,l,method_verb
"sh" "/testmethod.sh"
set,s0,l,method_type
program
set,s0,l,trace_launch
1
save,s0,l
-------------------------------------------------------
Create file testmethod.sh in /tmp directory. Contents of testmethod.sh is as follows:
-------------------------------------------------------
#!/bin/sh -xvf
# print date time stamp to output file
date > /tmp/testmethod_output.txt
------------------------------------------------------
Now connect to docbase as non-super user "tuser1"
Generate a login ticket for "tuser1" (for himself)

API> getlogin,c
...
DM_TICKET=AAAAAgAAAOQAAAABAAAAAkcxy85HMcz6AAAAOHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAHR1c2VyMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHJycHZtaW5kZXgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAFJSUFZNSU5ERVgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAG1hbmFnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaUkzL25xYXZPZzdMWEY4WWVNajg0dUw0R0tzcG94TVZkYW9sdnFHcmZzaFJ6VmUzNTFPRmJBPT0=

Dump the login ticket to see ticket detail

API>dumploginticket,c,DM_TICKET=AAAAAgAAAOQAAAABAAAAAkcxy85HMcz6AAAAOHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAHR1c2VyMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHJycHZtaW5kZXgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAFJSUFZNSU5ERVgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAG1hbmFnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaUkzL25xYXZPZzdMWEY4WWVNajg0dUw0R0tzcG94TVZkYW9sdnFHcmZzaFJ6VmUZNTFPRmJBPT0=
...
LOGIN TICKET DUMP
==========================================
Version : 5.3 (ticket version 2)
Scope : global
Sequence Number : 0000000002
Single Use : No
Create Time : Wed Nov 07 06:29:34 2007
Expiration Time : Wed Nov 07 06:34:34 2007
User : tuser1
Password : *********
Domain : rrpvmindex01
Server : vmcs01_535_ora10g203
Docbase : vmcs01_535_ora10g2
Host : RRPVMINDEX01
API> quit

Quit out of API. This is to make sure the next IAPI DMCL connection pool does not contain any "tuser1" entries.
Now run IAPI and connect as super user, say "dmadmin".

Then connect to docbase as "tuser1" using the login ticket generated from the previous IAPI run.
API>connect,vmcs01_535_ora10g203,tuser1,DM_TICKET=AAAAAgAAAOQAAAABAAAAAkcxy85HMcz6AAAAOHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAHR1c2VyMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHJycHZtaW5kZXgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAHZtY3MwMV81MzVfb3JhMTBnMjAzAAAAAAAAAAAAAAAAAFJSUFZNSU5ERVgwMQAAAAAAAAAAAAAAAAAAAAAAAAAAAG1hbmFnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaUkzL25xYXZPZzdMWEY4WWVNajg0dUw0R0tzcG94TVZkYW9sdnFHcmZzaFJ6VmUzNTFPRmJBPT0=
...
s1
After new session is established. Run "apply DO_METHOD" to launch "TestMethod".

API> apply,s1,,DO_METHOD,METHOD,S, TestMethod
...
q0

Check the "/temp" directory, see if output file testmethod_output.txt is created successfully, and the output file "testmethod_output.txt" contains the date timestamp info.

Run the following commands to make sure the "TestMethod" is run successfully.
Make sure you do NOT see errors like this:
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES
result : 0
process_id : 0
launch_failed : T
method_return_val : 0
os_system_error : No Error Message Available
timed_out : F
time_out_length : 60
SYSTEM ATTRIBUTES
APPLICATION ATTRIBUTES
INTERNAL ATTRIBUTES
API> getmessage,s1,3
...
[DM_METHOD_E_ASSUME_USER_UV]error: "Your method named (Method2) failed to execute because the assume user process could not validation your user credentials. Assume User Process returned (-11=DM_CHKPASS_BAD_LOGIN)."
API>

If testing proves it is a ticket timeout issue.
This issue will be resolved in CS 5.3 SP6
Since this support Note was written prior to reslease of SP6, please request a eng hot fix

CS_5.3_SP5_BUG_148355_WINDOWS_ORACLE_HOTFIX.zip
CS_5.3_SP5_BUG_148355_WINDOWS_SQL_HOTFIX.zip
ContentServer_aix_oracle_5.3SP5_bug_148355.tar.gz
ContentServer_solaris_oracle_5.3SP5_bug_148355.tar.gz

Please provide the following information:
1. Content server version.
2. OS platform and version.
3. RDBMS info.
4. Test results testing above.