As for testing without a gateway you’re in luck, Hylke posted the following in Slack today:
As we just updated all public routing servers to
v2.5.0, I thought it would be nice to let you guys know what new features we added to the backend in the last month, so here’s the summary
- …
- Sending simulated uplink to the Handler for testing applications or integrations
Some of these features still have to be implemented in the console, but you can already use them by talking directly to the APIs
ah nice, will check it out
Ik have played a bit with the Http Integration, and I used this code in a post.php file:
<?php
$postdata = file_get_contents('php://input');
file_put_contents('d:\removedforsecurity.com\www\loradata\data.txt',
$postdata . PHP_EOL, FILE_APPEND);
echo 'ok';
?>
It will write your data to a txt file on your server. The URL to this file needs to be configured in the Http Integration, of course! I didn’t have time yet, to figure out the SQL queries.
Good luck.
ok, some steps further. On advice of Hylke simulated input via the payload decode function:
var decoded = {"v1":12,"v2":13,"t":22,"h":88,"la":005,"lo":055};
return decoded;
And I trigger a payload to be delivered via ttnctl:
ttnctl devices simulate 201612001 11111111
payload is then sent to a ttnpost.php page I made (honoustly looking up most of it via google…):
<?php
//Make sure that it is a POST request.
if(strcasecmp($_SERVER['REQUEST_METHOD'], 'POST') != 0){
throw new Exception('Request method must be POST!');
}
//Make sure that the content type of the POST request has been set to application/json
$contentType = isset($_SERVER["CONTENT_TYPE"]) ? trim($_SERVER["CONTENT_TYPE"]) : '';
if(strcasecmp($contentType, 'application/json') != 0){
throw new Exception('Content type must be: application/json');
}
//Receive the RAW post data.
$content = trim(file_get_contents("php://input"));
//Attempt to decode the incoming RAW post data from JSON.
$decoded = json_decode($content, true);
//If json_decode failed, the JSON is invalid.
if(!is_array($decoded)){
throw new Exception('Received content contained invalid JSON!');
}
//Process the JSON.
// Create connection
$servername = "localhost";
$username = "username";
$password = "password";
$dbname = "dbname";
$myConnection= mysqli_connect($servername,$username,$password,$dbname) or die ("could not connect to mysql");
$vBattery1 = $decoded['payload_fields']['v1'];
$vBattery2 = $decoded['payload_fields']['v2'];
$temperature= $decoded['payload_fields']['t'];
$humidity= $decoded['payload_fields']['h'];
$locationLat= $decoded['payload_fields']['La'];
$locationLong= $decoded['payload_fields']['lo'];
$sqlCommand="INSERT INTO sensordata(vBattery1, vBattery2, temperature, humidity, locationLat, locationLong) VALUES('$vBattery1','$vBattery2','$temperature','$humidity','$locationLat','$locationLong')";
$query=mysqli_query($myConnection, $sqlCommand) or die(mysql_error())
?>
(this ttnpost.php works if I call it from another php page with a json in the POST)
When I now trigger it with ttnctl a new row is added to the DB but has no values… To be continued (hints are welcome 
).
EDIT: Now it works see post below
Jeroen
ok, found it (corrected in the post above). I got much more than just my sensor value JSON.
I was expecting at the website:
{"v1":12,"v2":13,"t":22,"h":88,"la":005,"lo":055}
But I got:
{"app_id":"201612001","dev_id":"201612001","hardware_serial":"***************","port":1,"counter":0,"payload_raw":"EREREQ==","payload_fields":{"h":88,"la":5,"lo":45,"t":22,"v1":12,"v2":13},"metadata":{"time":"2017-02-09T18:24:23.022746155Z"},"downlink_url":"https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/201612001/01?key=ttn-account-v2.************************"}
so needed to adapt the array from
$vBattery1 = $decoded['v1'];
to
$vBattery1 = $decoded['payload_fields']['v1'];
very nice 
Now waiting for my gateway to arrive and build the live node
…so meanwhile you can read about SQL injection and prepared statements. 
Also, to ensure the requests are really coming from TTN:
Set a value for the integration’s Authorization HTTP header (beware that some servers might only support the official Authorization: <type> <credentials> syntax, and might even ignore invalid values when using Basic Access Authentication or Digest Access Authentication; see references in my other answer below). Or add a custom header and a value†, and validate that in your code:
Or: add some authentication to the URL (something like ttnpost.php?myauth=my-super-secret)
Or: check the key in downlink_url against what you know is correct
Or: or use a random URL (like my-super-secret-h7dnt01kqq9.php)
† Added to this answer September 2017; the Authorization header and custom header were added to the integration somewhere after its initial release, and after the initial version of this answer, as posted February 2017.
hahahaha, yes indeed, already did some reading on that topic. Lot of knowledge to digest, one step at a time
and that is a good tip to check if it actually comes from TTN… I could check the TTN account that is hidden in the downlink URL?
thx
ps still having some issue though passing a decimal via the query to the db, also there some reading up and improvements to make…
Make sure you get decimals in the JSON payload. I don’t see any in your {"h":88,"la":5,"lo":45,"t":22,"v1":12,"v2":13}.
The JSON that is sent by TTN should send numbers, so should not have quotes around numeric values:
"payload_fields": {
"v1": 12.3,
"v2": 4.56
}
Check your payload functions if it looks like "v1": "12.3".
Make sure the database column allows for decimals, like MySQL DECIMAL(6,2) to store values from -9999.99 to 9999.99.
Use prepared statements; in your example above you’re passing strings to the database: VALUES('$vBattery1', '$vBattery2', ...), being VALUES('12.3', '4.56', ...). The database will very likely convert that to numbers if your database columns are numbers, but it’s still an unneeded conversion that might give you surprises. In your example, VALUES($vBattery1, $vBattery2, ...) would already be better (but is still insecure).
Yes, that’s what I meant. Or, to avoid losing any data when you change the keys: don’t validate the key when getting the data, but store the key in the database as well. This way you can filter later on.
ah, it was the declaration in the database, I selected DECIMAL but this has a default type of 10,0 … pfff
Made a new version with $_GET password, prepared statement and check for correct app__id and dev__id:
<?php
//check for correct password in url, URL needs to look like this : www.yoururl.com/ttnpost2.php?pwd=password
if ($_GET["pwd"] == "password")
{
//Make sure that it is a POST request.
if(strcasecmp($_SERVER['REQUEST_METHOD'], 'POST') != 0){
throw new Exception('Request method must be POST!');
}
//Make sure that the content type of the POST request has been set to application/json
$contentType = isset($_SERVER["CONTENT_TYPE"]) ? trim($_SERVER["CONTENT_TYPE"]) : '';
if(strcasecmp($contentType, 'application/json') != 0){
throw new Exception('Content type must be: application/json');
}
//Receive the RAW post data.
$content = trim(file_get_contents("php://input"));
//write post data to file for debug, you need to create data2.txt in the same folder as ttnpost2
//file_put_contents('data2.txt', $content . PHP_EOL, FILE_APPEND);
//Attempt to decode the incoming RAW post data from JSON.
$decoded = json_decode($content, true);
//If json_decode failed, the JSON is invalid.
if(!is_array($decoded)){
throw new Exception('Received content contained invalid JSON!');
}
// Create connection
$servername = "localhost";
$username = "username";
$password = "password";
$dbname = "dbname";
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
//fill variables from decoded JSON array
$celcius = $decoded['payload_fields']['celcius'];
$volt = $decoded['payload_fields']['volt'];
$latitude = $decoded['metadata']['gateways']['0']['latitude'];
$longitude = $decoded['metadata']['gateways']['0']['longitude'];
$app_id = $decoded['app_id'];
$dev_id = $decoded['dev_id'];
//validate app_ID and dev_id for security purposes before updating DB. You need to change these according to your own id'scandir'
if ($app_id == "boat" && $dev_id == "boat") {
//update DB using prepared statement
$stmt = $conn->prepare("INSERT INTO sensordata2(celcius, volt, latitude, longitude, app_id, dev_id) VALUES(?,?,?,?,?,?)");
$stmt->bind_param("ddddss", $celcius, $volt, $latitude, $longitude, $app_id, $dev_id);
$stmt->execute();
$stmt->close();
$conn->close();
}
}
else{
throw new Exception("Incorrect password");
}
?>
Let me know what you think…
and with this one you can test/check easily without needing input from TTN:
<?php
//API Url
$url = "www.yoururl.com/ttnpost2.php?pwd=password";
//Initiate cURL.
$ch = curl_init($url);
//Tell cURL that we want to send a POST request.
curl_setopt($ch, CURLOPT_POST, 1);
$jsonDataEncoded = "{\"app_id\":\"boat\",\"dev_id\":\"boat\",\"hardware_serial\":\"0004A30B001A4BC2\",\"port\":1,\"counter\":6854,\"payload_raw\":\"NdacWGYPaOmnH+LtowP5/xAAAgQB\",\"payload_fields\":{\"celcius\":15,\"volt\":4.02},\"metadata\":{\"time\":\"2017-02-09T20:51:08.72197317Z\",\"frequency\":868.1,\"modulation\":\"LORA\",\"data_rate\":\"SF12BW125\",\"coding_rate\":\"4/5\",\"gateways\":[{\"gtw_id\":\"eui-b827ebfffe817a05\",\"timestamp\":1452982252,\"time\":\"2017-02-09T20:51:08.6936Z\",\"channel\":0,\"rssi\":-119,\"snr\":-10,\"rf_chain\":1,\"latitude\":53.107,\"longitude\":6.125,\"altitude\":40}]},\"downlink_url\":\"https://integrations.thethingsnetwork.org/ttn-eu/api/v2/down/boat/boat?key=ttn-account-v2.*******************************\"}";
//Attach our encoded JSON string to the POST fields.
curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonDataEncoded);
//Set the content type to application/json
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/json"));
//Execute the request
$result = curl_exec($ch);
echo $result;
echo "<br>";
echo "done";
?>Note that application id’s are not secret; for device id’s I don’t know. For inspiration about goats see https://discovery.thethingsnetwork.org/announcements/handler
It seemed the first payload you posted had longitude and latitude from some GPS on the node? Now you’re using the location of the first gateway in the list (and I don’t know if the first has the best reception).
$stmt->bind_param("ddddss", $celcius, ...
Some debugging left there I guess
Don’t forget to store a timestamp too (or use a column that is automatically set on insert, like MySQL: received timestamp default current_timestamp).
Hi Arjan, again thx for the feedback. How would you come to know the app or dev id? But i now know how to do it so can use anything… the lat lon indeed comes from the gateway, it is not really relevant info i was just trying to get some data from deeper in the json.
But still good remark because now i understand why is called gatewayS; there can be more then 1 listed if in range
Not sure what you mean with some debugging to do, do you mean something is wrong in the sql part?
I was referring to the "ddddss" in your code (and hence supplying 7 values for 6 parameters).
Maybe i misunderstood how it works, but i thought the ddddss says next inputs are 4 doubles and 2 strings, so 6 definitions for the 6 inputs…?
Oh, totally my bad! I don’t know PHP, but it looked like garbage and I thought you deleted some columns before posting here… I almost deleted it from your post; glad I didn’t
Hi,
i hope I can get some help.
I followed the example above and I managed to get the payload out of JSON and into my database.
Now working on the security and I am stuck on trying to process the headers sent btythe ttn http integration.
I used https://requestb.in and I see all the headers there, including:
Host: requestb.in
Content-Length: 635
User-Agent: http-ttn/2.5.1
Content-Type: application/json
Accept-Encoding: gzip
Cf-Connecting-Ip:
X-Request-Id:
Cf-Visitor: {"scheme":"https"}
Via: 1.1 vegur
Cf-Ray:
Connect-Time: 0
Authorization: dummy_auth
Cf-Ipcountry: IE
Total-Route-Time: 0
Connection: close
add-Key: dummy_key
However, when I try to parse the headers in my PHP script using the function getallheaders() I only get a subset of the above, specifically:
Host: requestb.in
Content-Length: 635
User-Agent: http-ttn/2.5.1
Content-Type: application/json
Accept-Encoding: gzip
I don’t have much familiarity with servers and PHP so I don’t know whether the extra headers were stripped out by the server, not picked up by the getallheaders() function or something else.
Any advice would be appreciated.
Thanks
Riccardo
That value for Host cannot be true; it should of course be the domain of your very own server.
Most other headers you see with RequestBin are specific to their setup at Cloudfare. (Think: headers added by their load balancer, SSL off-loading, CDN.)
So, the HTTP headers you’re seeing on your own server are probably fine. [Not quite; see answers below.]
Why do you think you need more headers then?
From my post above: From the initial version (meanwhile edited) of my post above:
That does not did not refer to additional HTTP headers (which were not yet supported at the time of writing), but to URL query parameters and the content of the JSON payload.
Yes, I copied and pasted the host from the lines above when writing the questions. I get the right one in PHP.
With regards to the security I was trying to do it by the book and follow the ttn implementation for the http integration, using the
Authorization: dummy_auth and additional optional key:
add-Key: dummy_key.
However it appears difficult to do, so ?myauth=my-super-secret will have to do. Also I can use the json fields for further authentication.
Thanks for the advice
Riccardo
If you’re saying you provided values for Authorization and add-Key and got those in RequestBin but not in your own server, then that seems to be weird indeed.
(Next time please copy the exact headers.)
Will do.
Maybe someone could post the PHP code they use to extract, parse and use the headers?
It is good to start with something known to work.
Thanks for the advice
Riccardo