Flight-Data SQL Archive

Update: 10/6/2019  

The historical search at flight-data.adsbexchange is currently down until further notice.

ADSBx hopes to bring it back to the public with a more cost effective solution that is secure, monitored, and potentially a subscription model. If you are hosting a receiver sending data and use is not commerical, a feeder will get access at no cost ( usage within reason ).

We will investigate if ADSBx can extend the feeder historical search to 2+ years of data and maintain affordable infrastructure and storage.

There will be a hard cap on usage to prevent abuse by commercial organizations pretending to be enthusiasts.  Limits are the only way we can offer enthusiast tracking in its entirety for no cost.

Remember ADSBx already offers the following feeder access points.

  1. FREE unblocked and unfiltered graphical global tracking via Global UI Map
  2. FREE unblocked and unfiltered global REST API

Those two feeder features alone are not available on ANY ADS-B site.  And if you wanted to buy them in filtered data from FR24 or FlightAware – then you are going to be paying $100,000 a year.

 

Let’s learn by example!

Calculation for 1 day of data equivalent from ADSBx vs FlightAware worked out below.

1 request every 5 seconds = 17,280 per day at $0.002 per API request = $34.56 USD

But WAIT .. FlightAware charges 1 API request per every 15 aircraft returned.

If say ADSBx tracks on average 6,000 aircraft (peaking at 10,000+ and on  low end 5000). 

So, let’s price it like FlightAware.

6000 aircraft divided by 15 Aircraft per API request = 400 average API up charges per request

400 API up charges times 17,280 api calls a day times $0.002 per api call = $13,824 per day!

And that’s using the LOWEST COST for the FlightAware API.

Perhaps you should ask FlightAware for unfiltered and unblocked global data.  OH WAIT … You can’t! They filter and block data! They even filter the data they send back to your feeder!

Well at least they give you a FREE API?  .. What? … No? … hmmmm

How about global map with all tracked aircraft, including VIP, police, government, private, and military? … What?  … No? …. hmmmmm

You are getting a real ‘$89 value’ for that feed you are sending aren’t you?

 

Let’s get back to why you are reading this page.

Please remember the costs below do not include all the volunteer time and effort to make it work.  Nor do they include the occasional contract database admin to fix the rare issues that could not be corrected by ADSBx volunteers. Nor have I detailed any of the ingest infrastructure and cloud infrastructure that is running at data centers around the world.

The ADSBx historical search is routinely abused by hedge funds, commercial entities, and other for-profit entities. This incident is likely one of those cases, as over the last few months ADSBx has had many inquires from such entities.  The response was not positive when these entities were told that must support ADSBx for access to the data.  A few, ADSBx later discovered, were already using the data in violation of ADSBx terms.

After looking at usage patterns on AWS, it appears that someone may have decided to data mine / scrape the historical search starting in July.  They generated billions of read IOPS (by pulling full sets of uncached data).

Around the same time ADSBx was making headlines and we noticed the flight-data was running slow.  Thinking it was a temporary increase due to publicity, Cloudflare was placed in front and the PostgreSQL was scaled to compensate (this was a mistake). This increase in resources simply increased the rate at which usage could burst and sustain.  This snowballed into a massive AWS bill.

Why did this happen?  If your database server runs a query that needs to read or write data to the disk you will use I/O requests. AWS bills for those I/O requests.

We don’t believe that long term there is an AWS solution for ADSBx to store this data set at $0.10 per GB.  As you can see from the billing, the data set is growing by 4 Terabytes a month.

The ADSBx AWS normally functions on a ‘low’ cost r4.xlarge, but the storage size is ever increasing – which simply makes AWS unsustainable for ADSBx in the long term.  The bill for the storage will need continue to grow to accommodate new data in the Postgres database.

Let me make it clear that donations rarely exceed $1,000 USD per month across all of ADSBexchange.com.

 

August:  $12,281.56 AWS TOTAL

September: $13,158.96 AWS TOTAL