Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Troubleshooting HTTP errors in SAP Cloud Integration (Inbound)

vitor_lemberck
Employee
Employee
‎12-28-2022 8:37 AM

Introduction

The focus of the blog is to help you understand and analyze all the HTTP errors that happen during the connection from your sender system to your Cloud Integration (inbound). Also, this blog will provide some troubleshooting tips that will help better understand where exactly is the problem and how to avoid/prevent it in the future
To better understand this blog post, it is expected that you have some kind of knowledge of Cloud Integration Inbound configuration. It is advised to read SAP Help below, which has a great explanation of how each type of Inbound configuration works:

It is also advisable to read the Blog below (depending on whether your environment system is NEO or Cloud Foundry) for a better explanation of the configuration using Client Certificates:

 

Troubleshooting

After creating your scenario to connect to Cloud Integration, some types of HTTP errors may appear in the sender side. Each error has a few different causes, for example, the HTTP 404 error can be caused by a wrong endpoint configuration or an internal problem in the worker nodes of your system. To distinguish what is causing the error and how to solve it, we can use some logs and tips.
   

Necessary traces/logs

Cloud Integration has two logs that together can provide more detailed information about the problem. You can access this log from the Monitoring Dashboard. In the section Access Logs select the System Log Files tile. You get a list of log files with the most recent files at the top. There are one or more HTTP Access or LJS/Trace files with nearly the same time stamp, one for each runtime node started in your tenant. Because you don’t know at which runtime node the request arrives, you need to check all of those files. Later in the blog, I will give tips on which file to download to check for errors more precisely.

Note: The Default trace logs from Cloud Integration may be named LJS_trace.log or Trace.log (depending on whether your system is in NEO or Cloud Foundry) but they are the same logs.


HTTP Access Log:
For each inbound call reaching the Cloud Integration runtime node, one line is written into the HTTP Access Log containing the IP address of the calling system, the certificate or user contained in the request, the date, the path, and the HTTP return code. Through this blog, we can also be sure that the call has reached Cloud Integration. The line in the log file has the following format:

  • <IP Address> – <certificate or user> [<Date>] GET <path> HTTP/1.1 <return code>


LJS log or Trace log:
If the certificate is received, but there is an error during the authorization check, you should consult the LJS Trace (also known as CP Default Trace) for more details about the error. The formatting of the log that is written to this trace log is varied and can depend on each thread. But it usually has this format:

  • <Timestamp> #<Timezone>#<Severity>#<Class>##<user>#

Sample:

  • 2022 12 21 10:35:00#+00#ERROR#com.sap.it.nm.core.concurrent.lock.ClusterLock##anonymous#

Log Node ID
The log ID is what links an HTTP to an LJS. If your tenant has multiple runtime nodes you can use the ID of your HTTP logs to know which LJS/Trace log to use to check for more details of the error. Check the name of the HTTP Access Log file, where you found the request, to get the ID of the runtime node (LJS/Trace file). The name of the file has the following format: http_access_<Node ID>_<date>.log. The Node ID is the ID of the runtime node, that received the request. Search for the corresponding CP Default Trace with the same node ID, file name format: ljs_trace_<Node ID>_<date>.log.

 

Additional necessary information

In addition to the logs, the following information (in the Sample part) will help in analyzing the problem. The main one is the Path of your Iflow. The Path is the URL that your Iflow uses to identify external calls. It is unique to each Iflow and when called by some system it will start the Iflow. It is set in the sender adapter (parameter Address) of your iflow.

  1. Go to your Cloud Integration Monitor > Click in All tile of Manage Integration Content;
  2. Search for the Integration Iflow in Search Field;
  3. Check the Endpoint tab.


The path of our example iflow will be 'http/Inboudtestconnection'.

Analyzing the Problem

Now we have the necessary information from the Iflow where the error is happening (using the Path) and we know which logs to use. For better accuracy of the analysis, it is advisable that you reproduce the problem again (calling the iflow that is responding with the HTTP error) so you don't have to download so many logs to analyze.

If it is possible to reproduce the problem, check step number 1, if not, use step number 2.

    1. Copy the Iflow path and reproduce the call again, after the problem appears in the sender, download the latest HTTP Logs and LJS Logs from your Cloud Integration. If you have many logs with similar timestamps, download the last logs that were created after the timestamp that you reproduced the problem. For example, if you reproduced the problem at 15:34:00, download all HTTP and LJS logs that were created after that timestamp. If you only have one LJS and one HTTP log per day in your tenant, you can download just the logs for the day.
    2. Check the Iflow Path and copy this information. Use the timestamp of when the problem last occurred, and download all the logs that were created after the problem appeared.


Note:

    1. The logs are limited by day, i.e. if the problem occurred at 15:44:00 on December 21, and it cannot be reproduced, you can only download logs that occurred after that time and that were not created on or after December 22.
    2. The time shown in your System Log Files (and tenant) is the same timezone set in your computer. If the sender uses a different timezone than your computer, you need to do a conversion. For example, the sender shows that the error occurred at 19:44:00 (GMT +1) but your tenant and computer are in GMT -3 (15:44:00). In that case, you will have to download the logs from 15:44:00 and above.
    3. The timestamp of the files that are downloaded are in GMT/UTC+00 (see the sample part).
    4. The creation or update of the log can be seen in the Updated At tab of your Monitor System Logs Files:
      vitor_lemberck_0-1710862828643.png


Sample

In the example below, the Sender is calling the Iflow Test Connection Inbound, which has path /http/Inboudtestconnection. The problem in the sender was an HTTP 404 - Not Found that occurred for two minutes, and the problem cannot be reproduced. The problem happened at 5:01:50 (GMT -3) on the 21st of December (which is the same timezone as my computer/tenant).

  • Path: /http/Inboudtestconnection
  • Timestamp: 5:01:50 (GMT -3)
  • Error: HTTP 404 - Not found

Looking at the System logs, I see that there are 4 potential logs to download because they were created or modified after the problem occurred (logs are from the same day and are above the hour and minutes when the problem occurred). Two of them are LJS logs and two HTTP logs:
After downloading the 4 files, I open them in Notepad++ (or some text editor of your choice) and search for the path /http/Inboudtestconnection, using the Find All in All Opened Documents option:


The search returns me three results in one of the two HTTP logs used. There are three lines in them, two that were successful (HTTP 200) and one with the HTTP 404 error at the same timestamp as when the problem occurred in the sender:


At number one in the screenshot above, we see the HTTP Log  Node ID. We will use it to know which LJS/Trace log is the correct one to check for the problem. The Node ID in question is 1b64592, so we will use the LJS log with ID 1b64592 (number 3 in the screenshot).
In number two we see the problem that happened (in timezone GMT/UTC+00), and we will copy the timestamp of the problem to check the root cause in LJS:

  • <IP> (<IP>) - - [21/Dec/2022:08:01:50 +0000] GET /http/Inboudtestconnection HTTP/1.1 404 5 78 <Tenant ID> -iflmap.hcisb.int.sap.eu2.hana.ondemand.com:443 -

Searching for timestamp 08:01:50 in the LJS log with ID 1b64592, we can see that the root cause of the problem is the error:

  • 2022 12 21 08:01:50#+0000#ERROR#com.sap.it.rt.adapter.http.servlet.HttpMessageServlet#anonymous#http-nio-8080-exec-6#com.sap.it.rt.adapter.https.http.component#na#na#na#na#No consumer to service request com.sap.hcp.cf.logging.servlet.filter.LoggingContextRequestWrapper@640ec0af#95837231-f4a7-4eaf-6d81-c4d96dddbec5#10.0.79.104#0

When searching for error 'No consumer to service request' in SAP KBAs (Launchpad), we found the KBA below that explains the problem and how to solve it:

  • 3266623 - 'No consumers available on endpoint' error in SAP Cloud Integration

Tips

    1. If you are calling Cloud Integration but no call shows up in your HTTP logs (Cloud Integration side), the problem is usually with the sender. Usually, it is a connection or firewall problem on the sender system side and should be checked on the sender system. You can use ICM trace or HTTP trace from sender side to check what is causing the issue.

    2. Most of the time when facing an HTTP error (inbound) the Cloud Integration will not generate any messages in the Monitor. But if a message is generated, check the error that the message shows in the Monitor and it is possible that the problem is in the Cloud Integration connection to the Receiver. Often this occurs when the scenario is Sender -> Cloud Integration -> Cloud Connector -> On-premise system or when the response from Cloud Integration is HTTP 500 - Internal Server Error.

    3. When checking the HTTP + LJS/Trace logs, most of the time the exception appears, showing what the real problem is (lack of roles, Load Balancer certificate is not trusted, etc). And you can search for documentation (in Launchpad) that mentions the found error to know exactly the cause and the solution.

    4. If the error is only shown in HTTP but nothing is logged in LJS/Trace you should check the HTTP Access Log part of the Blog below:
    5. In scenarios that use the AS2 Adapter (sender) the error is detailed and almost always logged in the LJS with a lot of information. Like 'Load Balancer Certificate <Certificate> not found' or 'Configuration error in <Parameter name> parameter' and this is very useful for analyzing the problem in AS2 scenario.

 

HTTP errors

Below is a description of the most common causes and most common checks for HTTP errors and their respective error codes to use more quickly in case you decide not to use the analysis described above or did not find the cause of your problem in the LJS/Trace log.

401 - Unauthorized
The configuration required to authenticate the call is dependent on several factors, including what type of authentication you are using and whether the Cloud Integration tenant is hosted on NEO or Cloud Foundry.

Using Basic Authentication
You need to verify that the Sender Adapter is configured to use Basic Authentication, ie. the Authorization is set to 'User Role'. You also need to ensure that the Sender system is passing a Valid User ID in the call. Note that technical S-Users are not supported for use with Cloud Integration,  see:

  • 2974137 - Technical S-User receives 401 response in Cloud Integration.

Using Client Certificate Authentication on NEO
You need to ensure that all configuration steps outlined in the following SAP Community Blog have been implemented fully:

Specifically, you should verify the following points:

  • The Sender Trust Store must contain the Key Pair being used to authenticate the call
  • The Root Certificate of the Load Balancer has been uploaded to the Trust Store in the Sender system
  • This Key Pair must be signed by a CA which is supported by the Load Balancer. For more information on supported CAs, see Load Balancer Root Certificates Supported by SAP.
  • The Sender Adapter is configured to use Client Cert Authentication (either directly in the iFlow or using role based authorization with Certificate to User mapping implemented)

Using Client Certificate Authentication on Cloud Foundry
You must ensure that all configuration steps outlined in the following SAP Community Blog have been implemented in full:

Take special care to ensure the following is in place:

  • The Root Certificate of the Load Balancer has been uploaded to the Trust Store in the Sender system
  • The Sender Trust Store must contain the Key Pair being used to authenticate the call
  • This Key Pair must be signed by a CA which is supported by the Load Balancer. For more information on supported CAs, see Load Balancer Root Certificates Supported by SAP.
  • The client certificate used for the inbound call needs to be maintained in the service key.
  • If you are using the deprecated option to configure the client certificate directly in the integration flow, the client certificate is required additionally to be maintained in the service instance.


403 - Forbidden
This indicates that the call is failing due to missing permissions on the Cloud Integration side. If you are using Basic Authentication, or you are using Client Certificate Based Authentication with the 'Role Based Authentication' method described in either of the above blogs, then within the adapter configuration you will see the 'User Role' parameter defined in the adapter. By default, this is ESBMessaging.send. You need to ensure that the user being used to authenticate the call (or the user which is mapped to the certificate) has this role assigned to it.

Additional Note for Troubleshooting - If attempting to troubleshoot any of the above issues on a NEO Cloud Integration Tenant, it may be useful to check the inbound calls in the HTTP Access Logs. These are not available for Cloud Integration tenants on Cloud Foundry.

For every inbound call to the tenant, a line will be written to the log and this contains important information such as the IP address of the calling system, the timestamp, the path being called, and most importantly (for troubleshooting the above issues), the HTTP response code and the user or certificate which has been passed to the tenant to authenticate the call. This typically has the following format -

  • <IP Address> – <certificate or user> [<Date>] GET <path> HTTP/1.1 <return code>

If the portion of the written message where the certificate or user would be expected to be displayed is blank, this would indicate that either this is not being sent by the Sender system or, the CA which has signed the certificate is not supported by the Load Balancer (and is getting blocked from being passed to the tenant).

404 - Not Found
The endpoint does not exist or is unavailable. Log into the CPI tenant and check the status of the iFlow in the 'Manage Integration Content' Monitor. Is the iFlow deployed and showing in a 'Started' state? Does the endpoint listed under the 'Endpoints' tab match the URL which you are calling exactly? If yes this can be caused by Outage or Internal problem:

  • 3266623 - 'No consumers available on endpoint' error in SAP Cloud Integration


500 - Internal Server Error
The HTTP 500 error can be caused by an internal problem in your Cloud Integration or a problem when processing the message on the receiver (endpoint). If you receive the HTTP 500 error in the Sender when calling Cloud Integration, but a message is generated in your Cloud Integration Monitor, the error is usually caused between Cloud Integration and the Receiver, and the cause can be checked in the message generated (in Cloud Integration Monitor). If you are getting the HTTP 500 error but no message is generated in Cloud Integration side, it is possible that the problem is occurring in the worker nodes of your Cloud Integration, as seen in the SAP KBA:

  • 3280549 - 'Servlet.service() for servlet' and 'HTTP 500' errors in Cloud Integration


502 - Bad Gateway
This error normally occurs when the amount of messages sent to Cloud Integration at the same time is too much that the tenant with the current resources takes. This error can be seen in the SAP KBA below:

  • 3248977 - 502 inbound error when large amount of messages sent

 

Conclusion

By analyzing the HTTP log and the LJS logs you can be more precise about where the problem is occurring and the root cause. And when you find the error you can look for documentation on what caused it and how to fix it. These two logs provide a deeper analysis than just an HTTP code and can help you a lot to analyze connection errors with your Cloud Integration.

From all my experience with SAP Cloud Integration support, most HTTP 401 and HTTP 403 errors are caused by misconfiguration. HTTP 404 errors can be caused by configuration or worker node errors and HTTP 500 errors can be caused by receiver or Cloud Integration tenant internal error.

For info regarding Inbound configurations, you can check:

Another related SAP Documentation which might be informative about this topic:

Note: All screenshots in this Blog were captured in my Cloud Integration test system and in the Notepad of my machine.

10 Comments
Labels in this area
Popular Blog Posts