John Davidson

Message:

I am proxying the response of a cURL-Request in PHP like this:

$curl = curl_init($url);



curl_setopt($curl, CURLOPT_HEADER,         true);

curl_setopt($curl, CURLOPT_RETURNTRANSFER, false);  // makes curl_exec echo the response directly



header_remove();

$res = curl_exec($curl);

curl_close($curl);



if ($res) die();

My problem is that before cURL outputs the of the response (including the curl response header), the new line is sent that separates the header of my reponse from the body.

Example:

Remote's reponse is:

HTTP/1.1 200 OK

Date: Thu, 08 Apr 2021 15:14:55 GMT

Server: Apache/2.4.41 (Ubuntu)

Cache-Control: no-cache, private

Content-Length: 271

Content-Type: application/json



{"some":"json"}

The reponse of my script is:

HTTP/1.1 200 OK

Server: Apache/2.4.41 (Ubuntu)

Vary: Accept-Encoding

Content-Length: 446

Content-Type: text/html; charset=UTF-8



HTTP/1.1 200 OK

Server: Apache/2.4.41 (Ubuntu)

Cache-Control: no-cache, private

Content-Length: 271

Content-Type: application/json



{"some":"json"}

What I want is to use the remote's response a my own.

Of course I could

header('Content-Type: application/json');

but I don't want to only be able to proxy something that responds JSON.

Also I could

curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

and parse the response's header for the Content-Type but in that case I would not be able to forward large responses that exceed my memory and it would increase the latency of my script a lot because I could not start sending my response before cURL completed fetching the remotes response.

SOLUTION

Thanks to Barmer I got it working like this:

$curl = curl_init($url);



curl_setopt($curl, CURLOPT_HEADER,         false);  // IMPORTANT

curl_setopt($curl, CURLOPT_RETURNTRANSFER, false);

curl_setopt($curl, CURLOPT_WRITEFUNCTION,

  function($curl, $content)

  {

    if (!headers_sent()) header('Content-Type: '.curl_getinfo($curl, CURLINFO_CONTENT_TYPE));

    echo $content;

    return strlen($content);

  }

);



header_remove();

$res = curl_exec($curl);

curl_close($curl);



if ($res) die();

Explanation:

First I needed to change CURLOPT_HEADER to false.

Then I needed the CURLOPT_WRITEFUNCTION which is called for every chunk that is echoed by cURL. That is why I only add to the header when they are not sent.
I pumped an ISO of Ubuntu through it an it does not cache the content.

Next I will investigate the chunksize and if I can set it myself or at least how cURL decides the it.

Edit: Chunksize in my case is always 16384 bytes.

Edit: I can confirm, that a >2GB file downloaded with this method does not differ from the same file downloaded with wget.

---

My Answer:
To forward the cURL response header in PHP without using CURLOPT_RETURNTRANSFER, you can use the following code snippet:

php
$curl = curl_init($url);

curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, false);

ob_start(); // Start output buffering

$res = curl_exec($curl);

$header_size = curl_getinfo($curl, CURLINFO_HEADER_SIZE);
$header = substr($res, 0, $header_size);
$body = substr($res, $header_size);

header('Content-Type: application/json'); // Set the content type of the response
echo $header; // Output the response header
echo $body; // Output the response body

curl_close($curl);

ob_end_flush(); // Flush the output buffer


This code snippet uses output buffering to capture the cURL response and then separates the header and body of the response. It then outputs the response header and body separately using the `header()` function and `echo` statements.