HTTP integration

[rmeldo]

Hi,

i hope I can get some help.

I followed the example above and I managed to get the payload out of JSON and into my database.

Now working on the security and I am stuck on trying to process the headers sent btythe ttn http integration.

I used https://requestb.in and I see all the headers there, including:

Host: requestb.in
Content-Length: 635
User-Agent: http-ttn/2.5.1
Content-Type: application/json
Accept-Encoding: gzip
Cf-Connecting-Ip: 
X-Request-Id: 
Cf-Visitor: {"scheme":"https"}
Via: 1.1 vegur
Cf-Ray: 
Connect-Time: 0
Authorization: dummy_auth
Cf-Ipcountry: IE
Total-Route-Time: 0
Connection: close
add-Key: dummy_key

However, when I try to parse the headers in my PHP script using the function getallheaders() I only get a subset of the above, specifically:

Host: requestb.in
Content-Length: 635
User-Agent: http-ttn/2.5.1
Content-Type: application/json
Accept-Encoding: gzip

I don’t have much familiarity with servers and PHP so I don’t know whether the extra headers were stripped out by the server, not picked up by the getallheaders() function or something else.

Any advice would be appreciated.

Thanks
Riccardo

[arjanvanb (Arjan)]

@rmeldo

when I try to parse the headers in my php script using the function getallheaders() I only get a subset of the above, specifically:

Host: requestb.in
…

That value for Host cannot be true; it should of course be the domain of your very own server.

Most other headers you see with RequestBin are specific to their setup at Cloudfare. (Think: headers added by their load balancer, SSL off-loading, CDN.)

So, the HTTP headers you’re seeing on your own server are probably fine. [Not quite; see answers below.]

@rmeldo

I managed to get the payload out of JSON and into my database

Why do you think you need more headers then?

From my post above: From the initial version (meanwhile edited) of my post above:

@arjanvanb

Also, to ensure the requests are really coming from TTN: I’d add some authentication to the URL (something like tttpost.php?myauth=my-super-secret), or check the key in downlink_url against what you know is correct, or use a random URL (like my-super-secret.php).

That does not did not refer to additional HTTP headers (which were not yet supported at the time of writing), but to URL query parameters and the content of the JSON payload.

[rmeldo]

Yes, I copied and pasted the host from the lines above when writing the questions. I get the right one in PHP.

With regards to the security I was trying to do it by the book and follow the ttn implementation for the http integration, using the
Authorization: dummy_auth and additional optional key:
add-Key: dummy_key.
However it appears difficult to do, so ?myauth=my-super-secret will have to do. Also I can use the json fields for further authentication.
Thanks for the advice
Riccardo

[arjanvanb (Arjan)]

If you’re saying you provided values for Authorization and add-Key and got those in RequestBin but not in your own server, then that seems to be weird indeed.

(Next time please copy the exact headers.)

[rmeldo]

Will do.

Maybe someone could post the PHP code they use to extract, parse and use the headers?
It is good to start with something known to work.

Thanks for the advice
Riccardo

[Charles]

For this (and any IoT action) I would use node red doing something as simple as that

Check this link

Decode function should be something like that

var devID;
var dat;
devID = msg.dev_id;

// Decode data received my nodes are loranode01 or loranode02, ..
if ( devID.match(/loranode\d{2}/) ) {
    console.log("Valid devID=%s", devID);
    dat = decode_loranode(msg.payload_raw);
    console.log("bat=%d temp=%d hum=%d", dat.battery, dat.temperature, dat.humidity);

    strSQL = "INSERT INTO weather (NODE, TEMPERATURE, HUMIDITY, BATT, FRAMECOUNT) "
    strSQL += "VALUES ( ";
    strSQL += dat.devID + ", ";
    strSQL += dat.temperature + ", ";
    strSQL += dat.humidity + ", ";
    strSQL += dat.battery + ", ";
    strSQL += dat.fc + ", ";
    strSQL += ") ";
    console.log("MySQL=%s", strSQL);
        msg.payload = strSQL;
    return msg
} else {
    console.log("Invalid devID=%s", devID);
}

function decode_loranode(data) {
  var bytes = new Buffer(data, 'base64');
  var battery = (bytes[2] << 8) | bytes[3] ;
  var fc = (bytes[6] << 8) | bytes[7] ;
  var temperature = ((bytes[10] << 8) | bytes[11]) / 10;
  var humidity = bytes[14] / 2;
  console.log("batt=%d fc=%d temp=%d hum=%d", battery, fc, temperature, humidity);
  return {
    battery: battery,
    framecount: fc,
    humidity: humidity,
    temperature: temperature};
}

[arjanvanb (Arjan)]

@Charles

strSQL = "INSERT INTO weather (NODE, TEMPERATURE, HUMIDITY, BATT, FRAMECOUNT) "
strSQL += "VALUES ( ";
strSQL += dat.devID + ", ";

…ai, what if I figure out the URL you’re using, and then send a value for fc that is something like 1); drop table weather; --…?

[arjanvanb (Arjan)]

@rmeldo

using the
Authorization: dummy_auth and additional optional key:
add-Key: dummy_key

The Authorization header not being visible might be a known issue on some servers, if the server only supports Basic/Digest authorization, or is strict about the Authorization: <type> <credentials> header format:

In order to get HTTP Authentication to work using IIS server with the CGI version of PHP you must edit your IIS configuration “Directory Security”. Click on “Edit” and only check “Anonymous Access”, all other fields should be left unchecked.

…and:

Workaround for missing Authorization header under CGI/FastCGI Apache:

SetEnvIf Authorization .+ HTTP_AUTHORIZATION=$0

Now PHP should automatically declare $_SERVER[PHP_AUTH_*] variables if the client sends the Authorization header.

(It’s an old reference and I’ve not read any of the other comments.)

Also, on Stack Overflow Mun Mun Das writes:

Authorization header […] is discarded by apache if not in valid format. Try using another name.

…to which Mika Tuupola responded:

AFAIK Apache will accept any format Authorization header. It is PHP which discards anything else than valid Basic or Digest header.

Either way, surely “dummy_auth” is not a valid value as it does not follow the syntax Authorization: <type> <credentials>, such as Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l .

All this would not explain why add-Key would be missing, if you’ve indeed configured that too.

(@rmeldo, meanwhile I understand that nowadays the HTTP Integration supports setting an Authorization header or custom header, which it did not support in its initial release. I’ve edited my earlier answers for future readers.)

[Charles]

@arjanvanb
correct, but it was just a sample to help I do not have any table, and my sql instance is firewalled only with my admin IP
but anyway no sql admin would allow internet user to delete data or tablel not ?

[arjanvanb (Arjan)]

@Charles

my sql instance is firewalled only with my admin IP

Though your link refers to using MQTT to subscribe to a TTN server from Node-RED, in the context of the HTTP Integration TTN (or anyone) would be able to post data to Node-RED. With bad data, Node-RED would then craft a SQL statement one might not expect, no matter if the database can be accessed directly from the Internet or not.

@Charles

no sql admin would allow internet user to delete data or tablel not ?

Of course, SQL injection is also about exposing data…

There’s never an excuse for a dangerous example!

[rmeldo]

Thanks for the tips.
I will work my way through and report back.
Riccardo

[Charles]

@arjanvanb
Sorry my fault, the Node Red TTN in my stream was over MQTT not HTTP, since the goal was to insert into mySQL data from TTN. Even if original question was with http integration worth mentioning other simple solution can work.

By the way, I do not consider my example as bad, since data (payload) is parsed by a function in between MQTT and mySQL Insert. Each payload byte is decoded to be transformed by number so whatever you push in to the function, I think you would never be able to do any SQL command injection with dangerous commands in this example

Afterward of course the transfer between the output of Node Red to MySQL is relevant to network security and all rules applies (authentication, encryption, firewalling, …)

[HrMansen]

I found your code, while working on a PHP script, that react on a uplink message (from HTTP Integration) and making a downlink message to the device.
I’ve tried to use your code, to make my webserver, though Curl, make a Downlink-message.
But i do not get any result and no error.
Could i ask you (or maybe someone else ) to make an example, how to do this, or maybe point out my errors in this code:

<?php
//API Url (Got it from Uplink http-Integration, removed the key for this public forum)
$url = "https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/hrmansen1/http?key=ttn-account-v2.[REMOVED]";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
$jsonDataEncoded = "{\"dev_id\":\"hrmansenuno\",\"port\":1,\"confirmed\":false,\"payload_raw\":\"00\"}";
curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonDataEncoded);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/json")); 
 
//Execute the request
$result = curl_exec($ch);
echo "Result:<br/>";
echo $result;
echo "<br>";
echo "Errorcode: ";
echo curl_error($ch)
?>

[arjanvanb (Arjan)]

@HrMansen

\"payload_raw\":\"00\"

That doesn’t look like being Base64 encoded?

[HrMansen]

It isn’t

• i’ll try to change that to “MDA=” when i get home.

Thanks…

[HrMansen]

Unfortunately it did’nt change anything. I changed the line for the Json message to:
$jsonDataEncoded = "{\"dev_id\":\"hrmansenuno\",\"port\":\"1\",\"confirmed\":\"false\",\"payload_raw\":\"" . base64_encode("00") . "\"}";

But still get no result at the device, and no line in the Data log in the console.
Any ideas or php examples are welcome

[arjanvanb (Arjan)]

@HrMansen

\"port\":\"1\",\"confirmed\":\"false\"

And now these are no longer a number and boolean, like they were earlier, and like they should be. Doesn’t TTN return some error that shows in your echo statements? If not, try with curl -v on the command line first? See Downlink from HTTP integration does not show in TTN Console and does not reach my device and Using curl or Postman for the HTTP Integration on Windows.

Also, if this would work, you would send the text 00, which might not be what you want to send?

[HrMansen]

Hi Again.
I have experimented a little more since last time. I have found that my post date could be reduced to just dev_id and payload_raw.
I also found some example code from another project, and now i got my downlink message to be shown in the Data section of my device. But somehow my base64 encoding, ends with a extra 30 added to every digit
Just now while i am typing this, realizing that its the 30hex from the ascii character “0”

Thanks for the help!

Here’s my corrected code if anybody else is interested:

$data = array("dev_id" => "hrmansenuno","payload_raw" => base64_encode(chr(0) . chr(1)));
$url ="https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/hrmansen1/http?key=ttn-account-v2.rNA<redacted>YWiQ";

$post = json_encode($data);
echo $post;

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");  
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,$post);
echo " " .  curl_exec($ch);
echo "<br>Errorcode: " . curl_error($ch);
curl_close($ch);

[arjanvanb (Arjan)]

@HrMansen

somehow my base64 encoding, ends with a extra 30 added to every digit

Yes:

@arjanvanb

Also, if this would work, you would send the text 00, which might not be what you want to send?

@HrMansen

base64_encode(chr(0) . chr(1))

That probably indeed does the job, assuming you might also want to send much larger values, up tot 255 for each value. If you want to set “on” or “off”, then you can combine 8 states in a single byte. See https://www.thethingsnetwork.org/docs/devices/bytes.html

[loes7815]

Hi,
I’m not sure if I need to open a new topic, but all the above is also applicable for my question.
I’m trying to do the same, but I get stuck in the part before this topic starts;
I have webhook.site working, but now I can’t figure out what to fill in in the http integration window to make
$content = file_get_contents(“php://input”);
to work in my php-script.

I’m now locally working with xampp, so localhost. But i guess I need to place my php script somewhere on a server?
I have a Azure and visualstudio environment available (also where my database and API is). But dont know what is needed.
Thanks.