Enhancing Phoss-SMP URL Generation With Proxy Awareness

Introduction

Hey guys, let's dive into a common challenge faced when running phoss-SMP (Service Metadata Publisher) behind a reverse proxy, especially in environments like Kubernetes with Ingress. The main issue? Getting phoss-SMP to generate the correct public URLs when TLS (Transport Layer Security) terminates at the proxy layer. This article will explore the problem, the desired solution, and how a proxy-aware mode can significantly improve the situation. We'll also cover workarounds and a detailed explanation of how to reproduce the issue. So, buckle up, and let's get started!

Background / Context

The Problem: HTTP URLs in an HTTPS World

In many modern deployments, phoss-SMP operates behind a reverse proxy like ingress-nginx on platforms such as AKS (Azure Kubernetes Service). A typical setup involves TLS termination at the ingress, meaning the proxy handles the HTTPS connection with the client and forwards traffic to phoss-SMP over HTTP. This is a common and efficient practice, but it can lead to a tricky problem. When smp.publicurl.mode is set to request, phoss-SMP derives its public URLs from the incoming request seen by phoss. Because the upstream traffic is HTTP, phoss-SMP generates http:// ServiceGroup URLs, even if the original client request was over HTTPS. This mismatch causes headaches, especially when clients expect https:// URLs. In our case, this issue surfaces within a PEPPOL/eDelivery migration period where supporting both HTTP and HTTPS is crucial for interoperability.

To elaborate, the core of the problem lies in how phoss-SMP interprets the incoming request. With smp.publicurl.mode=request, the application directly mirrors the scheme and host from the upstream request. In a scenario where the proxy handles TLS termination, the upstream request is inherently HTTP. This results in ServiceGroup entries being published with insecure http:// links, despite the initial client request being secure https://. This discrepancy can lead to broken links, client confusion, and security concerns, particularly when dealing with sensitive data or critical business processes. The necessity for a solution that intelligently handles proxied requests becomes evident when considering the importance of maintaining secure and consistent communication channels.

Furthermore, this issue is not merely a cosmetic one. Many clients and systems are configured to strictly enforce HTTPS connections for security reasons. When they encounter http:// URLs in ServiceGroup entries, they may refuse to connect, leading to functional failures. The problem is exacerbated during migration periods, where systems need to support both HTTP and HTTPS for compatibility. A solution that dynamically adapts to the protocol used by the client is therefore highly desirable. The ability for phoss-SMP to recognize and respect the original client's protocol, even when operating behind a TLS-terminating proxy, is crucial for seamless and secure operations.

The Need for a Proxy-Aware Solution

Philip confirmed on Slack that a mode based on X-Forwarded-* headers would make sense, which prompted this issue. This highlights the importance of community feedback and collaboration in identifying and addressing these types of challenges. The current behavior necessitates a more intelligent approach to URL generation—one that considers the presence of a reverse proxy and uses standard headers like X-Forwarded-* to determine the correct scheme, host, and port. This leads us to the desired behavior, which aims to solve this problem elegantly.

Desired Behavior

Introducing a Proxy-Aware Mode

To tackle this, we propose introducing a new value for smp.publicurl.mode. Let's call it forwarded (though the name is open for discussion, guys!). This new mode should construct the external URL using the following precedence, leveraging standard X-Forwarded-* and Forwarded headers:

1. Scheme: Check X-Forwarded-Proto first. If not present, look for Forwarded: proto=. If neither is found, fall back to the request scheme.
2. Host: Check X-Forwarded-Host. If not present, look for Forwarded: host=. If neither is found, fall back to the request host.
3. Port: Check X-Forwarded-Port. If not present, derive it from the scheme (443 for HTTPS, 80 for HTTP).

This approach allows phoss-SMP to intelligently determine the original client's protocol and host, even when operating behind a proxy. If none of the X-Forwarded-* or Forwarded headers are present, the system should gracefully fall back to the current request behavior, ensuring compatibility with existing setups. This fallback mechanism is crucial for maintaining stability and preventing unexpected disruptions during the transition to the new mode. The hierarchical approach to header checking ensures that the most reliable information is used first, while still providing a safety net if headers are missing or improperly configured.

By prioritizing the X-Forwarded-* and Forwarded headers, the system can accurately reconstruct the original client's request, ensuring that generated URLs reflect the intended protocol and host. This is particularly important in complex network topologies where multiple proxies may be involved. The standardized nature of these headers also means that this solution is likely to be compatible with a wide range of proxy servers and load balancers. This makes the proposed forwarded mode a robust and versatile solution for deployments operating behind reverse proxies.

Benefits of the Proxy-Aware Mode

The benefits of this new mode are multifold. First and foremost, it resolves the issue of incorrect http:// URLs being generated in HTTPS environments. This ensures that clients receive the correct URLs, preventing connection errors and security warnings. Secondly, it simplifies the configuration of phoss-SMP in proxied environments. Administrators no longer need to resort to complex workarounds or hard-coded settings. The forwarded mode automatically adapts to the proxy configuration, reducing the risk of misconfiguration and making deployment more straightforward. Finally, this approach enhances the overall security posture of the system by ensuring that all URLs are consistent with the client's intended protocol.

Actual Behavior

The Current Problem in Detail

As it stands, with smp.publicurl.mode=request and TLS terminated at the proxy, ServiceGroup responses contain http:// URLs even when the client initially requested https://. This discrepancy arises because phoss-SMP only sees the upstream HTTP request from the proxy and is unaware of the original HTTPS connection. This behavior is not ideal, especially in environments where security and consistency are paramount.

The practical implications of this behavior can be quite significant. Clients expecting HTTPS connections may refuse to connect to the http:// URLs, leading to service disruptions. In some cases, clients may display security warnings to users, eroding trust and potentially discouraging use of the service. Furthermore, this issue can complicate the management of certificates and security policies, as administrators need to account for the discrepancy between the client-facing protocol and the internal protocol used by phoss-SMP. A solution that seamlessly bridges this gap is therefore essential for maintaining a secure and reliable system.

Steps to Reproduce

Replicating the Issue

Want to see this in action? Here’s how you can reproduce the issue:

1. Deploy phoss-SMP behind a reverse proxy/ingress that terminates TLS. This is a common setup in Kubernetes environments.
2. Set smp.publicurl.mode=request in your phoss-SMP configuration. This tells phoss-SMP to derive public URLs from the incoming request.
3. Call a ServiceGroup endpoint over HTTPS on the external host. Use a tool like curl or a web browser to make the request.
4. Observe that the returned ServiceGroup URLs start with http:// (the upstream protocol), not https://. You can inspect the response body to see the generated URLs.

By following these steps, you can easily verify the issue and appreciate the need for a proxy-aware mode. The simplicity of the reproduction steps highlights the commonality of this problem and underscores the importance of a robust solution.

Environment

Our Setup

For context, our environment is as follows:

• phoss-SMP: phelger/phoss-smp-sql:7.2.7
• Platform: AKS (Kubernetes)
• Ingress: ingress-nginx (TLS terminated at ingress; upstream HTTP)

This setup is quite typical for modern cloud deployments, making the issue we're discussing relevant to a broad audience. The combination of Kubernetes, ingress-nginx, and a containerized phoss-SMP instance is a common pattern, and the challenges encountered in this environment are likely to be shared by others.

Workarounds Considered

Evaluating Alternatives

We’ve considered a few workarounds, but they each have drawbacks:

• Forcing the ingress to speak HTTPS to phoss upstream (e.g., nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"): This requires phoss to expose an HTTPS port and adds complexity. It also might not be feasible in all environments.
• Hard-coding smp.publicurl.mode to a fixed base URL: This doesn’t handle dual HTTP/HTTPS during the migration period and is less flexible overall.

These workarounds highlight the need for a more elegant and adaptable solution. Forcing HTTPS upstream adds unnecessary overhead and configuration complexity, while hard-coding the base URL sacrifices flexibility and the ability to support mixed HTTP/HTTPS environments. The ideal solution should be transparent, adaptable, and require minimal manual configuration. This is precisely what the proposed proxy-aware mode aims to achieve.

The Limitations of Current Workarounds

To further elaborate on the limitations, forcing HTTPS upstream can introduce performance overhead due to the additional encryption and decryption processes. It also requires careful management of certificates and trust stores, adding to the operational burden. Hard-coding the base URL, on the other hand, creates a brittle configuration that is difficult to maintain and adapt to changing requirements. During migration periods, the ability to seamlessly support both HTTP and HTTPS is crucial, and a hard-coded solution simply cannot provide this flexibility. The proposed proxy-aware mode offers a more dynamic and robust approach, allowing the system to adapt to the actual client request protocol without requiring manual intervention.

Conclusion

Moving Forward

In conclusion, the current behavior of phoss-SMP when running behind a reverse proxy that terminates TLS can lead to incorrect http:// URLs being generated, causing issues for clients expecting https://. The proposed forwarded mode for smp.publicurl.mode offers a more robust and flexible solution by leveraging X-Forwarded-* and Forwarded headers to construct the correct public URLs. This approach ensures compatibility with common proxy setups and simplifies the configuration of phoss-SMP in these environments. By adopting this proxy-aware mode, we can enhance the overall security and reliability of phoss-SMP deployments.

Guys, I’m happy to test a snapshot build and provide feedback/logs. Thanks for reading!