Declassified: Killmail Scraping

A long time ago, I found Edzix Egdalds Video Series Adventures of Jaguar. In it, he uses one of the coolest methods to kill high-class site runners: Instead of relying on brute force to kill the typically remote repairing battleships, mostly Leshaks and Nestors. He just creates enough chaos for the site runners to die to the site on their own.

The ships of this trade are small, tanky afterburning ships like Jaguars or Punishers, paired with ECM. The plan is to hold down the site runners with the Jaguars/Punishers and spawn the Drifter while hindering remote repairs with the ECM from range. Even tho Leshaks have remarkable tracking, an afterburning Frigate orbiting at 500 meters has enough transversal that it can’t be hit (hit chance is smaller than 64bit float precision, so this is in fact a precise statement). There is still a few other threats for the frigates, but the chances to make it work are intact.

I really wanted to try this method, and one day I finally had luck and found a group of unsuspecting site runners behind a crit hole. It ended up working a little less than flawlessly (we booshed a Leshak onto our Kitsune, my Jaguar got killed by the site, while the other had no tackle fitted. Ooops) but we still managed to kill a few site runners. There was only one thing amiss. Where were the killmails?

Killmail mechanics

You probably have seen the popup “zkillboard doesn’t automatically get all killmails”. While killmail data can be fetched by anyone using a Killmail ID and Hash, these IDs and Hashes are only available for the Victim and the Pilot that got the final blow. If now one of these two guys decide to link their API to zkillboard (or their corp API is linked), then they get put in a database that anyone can access (depending on squizz’s mood there might be more or fewer ways to search them, but so far he always provided a simple summary of all the ones that he has received). This database currently holds about 60% of all generated killmails (Current Stats).

If an NPC gets the final blow, then a capsuleer inherits the killmail (it seems to be the guy with top damage) for almost all NPCs. The keyword here is almost. Because there is at least one exception to this rule: Vigilant Tyrannos, or the Drifter that the above-described method uses for almost all of the dealt DPS. As such, in our case, the two guys that got the killmail are a mysterious NPC and a player that really doesn’t like us at this moment.

So one of the less cool things in this method is that even tho you have caused the dismay of multiple billion worth in ISK of damage to another group with a small force, to the public it looks like you just fed a few frigates to a PvE fleet. Unless you can acquire that killmail in some other way. The easiest thing would probably be to convince some other, unsuspecting guy in the victim’s corp to slide over these killmails, but in our case, it was a pure alt corp and no unsuspecting guy can be found. It is also somewhat unreliable if you find such a guy and takes a lot of time.

Another avenue to take would be to guess the Killmail ID and Hash. But with the hash being pseudo-random and really long, this would take even longer. Unless… you know how the hash was made. Luckily enough, there is a guy on the eve forums that found out just that. And it turns out the hash only depends on a few things:

  • The Attacker (Easy, it’s the Drifter)
  • The Victim (Medium, just try all the Dudes you scrammed in your logs)
  • The Shiptype (Easy, look in log for who flew Nestor/Leshak)
  • The Exact second it died (Hard, possibily quite a few seconds possible, and no direct info in logs)
  • The Killmail ID (Hard, luckily these are increasing over time, you can kinda figure out a range with zkill that was used during the engagemnt)

With all that knowledge I got to work and wrote a simple script to try all these options 1 by 1. CCP also limits you to do 100 bogus requests a minute, so you have to slow your guessing down quite a bit. With handcrafted lists for each of these things, the process ends up being O(n) in the uncertainty in each of them. But because the Killmail ID and exact time it died are both dependent on time, one could also say that it takes O(n^2) with time uncertainty. And as this is the main thing you don’t know. In this first case, I had narrowed down the times of death to about a 30-second window, which resulted in about 3000 possible combinations for time only. If I only had 60 seconds of precision, this would already take 2 hours! But it worked! And we got our killmails!

Over time I had refined this method a little bit further. Instead of just limiting my rate of guessing, I had realized that CCP conveniently gave you an answer, to how many requests of your “error budget” you had used, and how long it takes until it was increased again. This meant that I could now go right to the edge of the error budget, from about 60 guesses a minute to just under 100 guesses a minute. In this version, the script was listed in the Opensource Spreadsheet/Scrips list for quite a while and ended up generating a few killmails. It however still took quite some time and knowledge to find a killmail, and I only know of one other guy that actually successfully used it.

The Potential Exploit

With all this coding and precise knowledge about killmails, over time a question arose. What if, you could not only find killsmails that you were part of, but also others? While it turns out that this just isn’t possible (if you knew at any moment exactly which players were online and guessed the ships they were flying, then it would only take about 1522 years to guess all the possible killmails – per Killmail ID and Second!).

However, a different Idea slowly took shape. What if, you could use this knowledge to keep killmails from going public? This seems pretty counterintuitive at first, but there might be a way that could actually do that. You see, ESI uses a cache for most things, so if you ask for a Killmail ID and Hash, then it is realistic that the response gets cached for some time, and any subsequent request yields the same result. So if you request a killmail with a valid ID and Hash before it actually happens, then you will get the answer that it doesn’t actually exist. And crucially anyone that happens to ask for the same killmail after you will get the same answer. Now at this point, you know that the killmail was actually valid, you can keep requesting it to keep it in cache until downtime at least.

So how do you request the killmail before it happens? The key here is that you don’t actually have to do it before it happens, just fast enough that you request it before it is in a database. And if you know how long it usually takes for some pod killmails to show up, then that might not even be that hard. So the game plan is like this.

  • On every tick, you find out who did damage to you last. He might be the guy that just killed you. You know your ship anyway, and you know the exact time it happened. Now you just have to guestimate the Killmail Id well enough and hope that it works.
  • If you actually end up dieing, then you can get the Id and Hash from the your API and request it again and again
  • Now if the enemy or you are linked to zkillboard, it will also try to get the killmail and fail because of the cache.

The main uncertainty is how exactly CCPs Cache works – and if this is actually possible. Looking at the ETAG data in a request reveals that while the Data that CCP sends over with a killmail is exactly the same, the “valid until” tag changes. Also looking at response times, it doesn’t seem to make any difference if you request the killmail for the first or hundredst time. That lead me to believe that there is in fact no cache over the whole thing, maybe only a cache over valid entries.

Last time I did something like this, I asked CCP and got a good “can’t tell you with certainty” so this time around I went ahead and actually tested id. While getting the time right +/- 2 seconds is somewhat doable (you just kill stuff and compare to your system time) guessing the Killmail Id is really hard. At the start, I just tried to linearly interpolate the number of killmails generated and added padding of +/- 50. But even that doesn’t really work, killmails really aren’t generated steadily. Sometimes in 5 minutes, there would be 120, other times 320 (and I tried during low activity AUTZ ofc). If you think about it, that makes a lot of sense. If a 30 man Harpy fleet runs into smartbombs somewhere, then that’s 60 extra killmails within seconds. In 5 minutes a fight can happen, or not happen. As such I just started guessing and trying over and over until – I requested a killmail before I actually generated it! And what happened? Well, ESI got me the killmail just fine. In other words, there is no cache, it doesn’t cache invalid requests, or there is simply no way that you can get the cache to be out of sync with the database. And I am happy that I don’t have to write an extensive EBR…

Revisiting the Code

With my experience in exact timing gained over the past fruitless attempt, I was now at least able to improve the brute-forcing algorithm. By coupling the Killmail Id and timestamp and only trying the exact values that don’t already exist in the killboard database the runtime now is effectively O(n) in aspect to time uncertainty. The improved code uses both binary search trees and newton iterations in combination to find your Killmail Id just from a single timestamp. This not only makes it a lot faster but also allows you to just put in a time that you think the killmail has been generated at the earliest. The updated code can be found here.

Stories along the way

Of course, there are always a few stories that go with such killmails. My favorite was a guy that ganked a Blockade Runner with an Arty Panther. Supposedly that Blockade Runner was transporting a courier contract with a 50b scam collateral, but the guy was smart enough to go empty first and see what would happen, before taking the scam contract with him. And immediately on the undock, he had to make a 90° turn to warp and got shot at.

Another interesting one is an Orca that went suspect in Hisec and got caught by The Wormhole Police, but the encounter wasn’t properly documented. Luckily I was able to fill in the papers afterward, call me the forensic department I guess…

And finally, I did rewatch all the Adventure of Jaguar videos and made sure that no Ship was left in the dark.

Volume 2 (+ Praxis + Nestor)
Volume 3 (+ Leshak)
Volume 4 (+ 2x Leshak + Nestor)

The Future

So you might ask yourself, do we see a future where anything will show up on zkillboard and we get to the transparent capsuleer? I am pretty sure we don’t, because there are just way too many options out there, even getting every killmail from a single character will be impossible. But DDing your own tackled carriers to hide the killmail might no longer be a sound strategy.

Leave a Reply