HTTP Listener Failed at Startup Possible Port Conflict on Port

Few days back I got an alert: Agent is Not Running. Possible port conflict on port(3872): Retrying the operation. Failed to start the agent after 1 attempts. Please check that the port(3872) is available.

Regarding this first check the status of the agent after that we will look into it.

[oracle@orahowracq1] cd /u01/app/oracle/product/12.1.0.2/agent/agent_inst/bin
[oracle@orahowracq1] ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Status agent Failure:unable to connect to http server at https://orahowracq1.hiw.com:3872/emd/lifecycle/main/. [peer not authenticated]
Agent is Not Running

Above we can see that agent is not running, so first try to start the agent normally as usually you do.

[oracle@orahowracq1] ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ................ failed.
HTTP Listener failed at Startup
Possible port conflict on port(3872): Retrying the operation...
Failed to start the agent after 1 attempts.  Please check that the port(3872) is available.
Consult emctl.log and emagent.nohup in: /u01/app/oracle/product/12.1.0.2/agent/agent_inst/sysman/log

During startup we get an error port conflict. Now you can check which process is listening to this port using simple combination of netstat with grep command.

[oracle@orahowracq1] netstat -anp |grep 3872 

Now find the PID of this process. So that we can mark it for kill.

[oracle@orahowracq1] ps -ef|grep agent

    root  6535     1   0   Jun 17 ?      0:02 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc
    root  3804     1   0   Jun 17 ?      206:39 /opt/VRTSobc/pal33/bin/vxpal -a actionagent -x
    root  6549  6535   0   Jun 17 ?      178:38 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc
    root  9184     1   0   Jun 17 ?      71:57 /opt/Navisphere/bin/naviagent -f /etc/Navisphere/agent.config
    root  8482  8071   0   Jun 17 ?      114:20 /opt/ecc/exec/mstragent -s
  oracle  15627 15547   0   Mar 13 ?      111:40 /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.2.0/jdk/bin/sparcv9/java -Xm
  oracle 24178  6501   0 23:22:46 pts/1  0:00 grep agent
  oracle 15547     1   0   Mar 13 ?      0:27 /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.2.0/perl/bin/perl

Make sure that process must be in: /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.2.0/jdk/bin/sparcv9/java. Finally kill the process having PID 15627 and start up the agent.

[oracle@orahowracq1] kill -9 15627
[oracle@orahowracq1] cd /u01/app/oracle/product/us/agent12c/bin
[oracle@orahowracq1] ./emctl start agent
[oracle@orahowracq1] ./emctl status agent
[oracle@orahowracq1] ./emctl clearstate agent
[oracle@orahowracq1] ./emctl upload 

Example:

./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ....... failed.
HTTP Listener failed at Startup
Possible port conflict on port(3872): Retrying the operation...
Failed to start the agent after 1 attempts.  Please check that the port(3872) is available.
Consult emctl.log and emagent.nohup in: /oradba/oemagent/product/12.1.0.2/agent/agent_inst/sysman/log

[oracle@orahowracq1] netstat -anp |grep 3872 
tcp   0      0 :::3872         :::*          LISTEN      7507/java

[oracle@orahowracq1 bin]$ ps -ef|grep agent

oracle    7507  7314  1 Apr14 ?        03:10:55 /oradba/oemagent/product/12.1.0.2/agent/core/12.1.0.2.0/jdk/bin/java -Xmx128M -XX:MaxPermSize=96M

[oracle@orahowracq1] kill -9 7507

Read more ...

ORA-3136: Inbound Connection Timed Out

Few days back we get an issue Inbound connection timed out (ORA-3136) in alert log. To troubleshoot this we need to dig into sqlnet.ora file and the alert log. 

The following four points are very important to troubleshoot this issue: 

• Check the alert log file and check from where the connection comes.
• Check the listener is up & running.
• Ping the server,make sure tnsping is working.
• Increase inbound_connect_timeout 

Regarding this we need to checked the value of "inbound_connect_timeout".

LSNRCTL> show inbound_connect_timeout

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))

LISTENER parameter "inbound_connect_timeout" set to 120

The command completed successfully

Here we can see that time is set to 120 which is ok.

The following would be the most likely reasons for this error :

1.Server gets a connection request from a malicious client which is not supposed to connect to the database.  In this case the error thrown would be the expected and desirable behaviour. You can get the client address for which the error was thrown in the sqlnet.log file that is local to the database.

2.The server receives a valid client connection request but the client takes a long time to authenticate more than the defined timeout.

3.The DB server is heavily loaded due to which it cannot finish the client logon within the timeout specified.

By default, the SQLNET.INBOUND_CONNECT_TIMEOUT is set to 60 seconds.

You can change the setting by adding the parameters SQLNET.INBOUND_CONNECT_TIMEOUT and INBOUND_CONNECT_TIMEOUT_<listener name>  to the $ORACLE_HOME/network/admin/sqlnet.ora file on the database server.

Setting the above parameters to a value of 0 implies an infinite time out.

Alternatively, you can use the lsnrctl command and issue the following:

LSNRCTL> set inbound_connect_timeout=<value>

Before opting for the above parameter changes, confirm that any firewall activity or Network Address Transalation (NAT) that may be occuring beween the client and and the database are not the cause of latency which is exceeding the timeout threshold.

 To identify the listener name and ORACLE_HOME and sqlnet.ora file we can use the below command.

[oracle@racdbq1.dcb.ichotels.com] ps -eaf|grep tns

Read more ...

ORA-01950: no privileges on tablespace users in Oracle

Few days back, an error was reported in the alert log which was due to no privileges on tablespace for auto execution of job as shown below.

ORA-12012: error on auto execute of job "ORAHOW"."DB_GROWTH_JOB"
ORA-01950: no privileges on tablespace 'ORA_DATA'
ORA-06512: at "ORAHOW.PROC_DB_GROWTH_INFO", line 4


After looking into it we found that user doesn't have privileges to allocate an extent in the specified tablespace. So using the below query we get the object type and the status of the object.

SQL> select owner,object_name,object_type,status from dba_objects where  object_name='DB_GROWTH_JOB';

OWNER          OBJECT_NAM      OBJECT_TYPE       STATUS
-------- ---------- ------------------------------------
ORAHOW      DB_GROWTH_ JOB      JOB            VALID

      

Now we will check the user default tablespace on which we need to give privileges.

SQL> select USERNAME, DEFAULT_TABLESPACE from dba_users where USERNAME='ORAHOW';

USERNAME                 DEFAULT_TABLESPACE
-----------------------------------------------
ORAHOW                     ORA_DATA

As you can see that user ORAHOW is using default tablespace ORA_DATA, but we also need to check the amount of space allocated for that user on that tablespace using the below query.

SQL> select TABLESPACE_NAME,USERNAME,MAX_BYTES,BYTES from dba_ts_quotas where USERNAME='ORAHOW';

no rows selected

Using the above query we can see that this user doesn't have access to this tablespace, so we will assign unlimited quota on this tablespace for this user.

SQL> alter user ORAHOW DEFAULT TABLESPACE ORA_DATA quota unlimited on ORA_DATA;

User altered.

If you want you can assign limited quota to that user.

ALTER USER <username> QUOTA 100M ON <tablespace name>
GRANT UNLIMITED TABLESPACE TO <username> 

Now will verify the above changes using the following query.

SQL>  select TABLESPACE_NAME,USERNAME,MAX_BYTES,BYTES from dba_ts_quotas where USERNAME='ORAHOW';

TABLESPACE_NAME        USERNAME         BYTES         MAX_BYTES
---------------------------------------------------------------
ORA_DATA                 ORAHOW          786432           -1
  

Finally, after assigning privileges on tablespace to the user we will execute the job.

SQL> exec DBREPORT.PROC_DB_GROWTH_INFO

PL/SQL procedure successfully completed. 

Read more ...