
How to Do Real Time Object Detection on Drones - sarthakjain
https://blog.nanonets.com/real-time-object-detection-for-drones/
======
funkaster
Title seems a little (maybe a lot?) misleading. It's not "on drones" it's
actually on a video stream coming from a drone. There's a huge difference:
you're doing all of the processing on a desktop/laptop PC.

There's some mention at the end of the article on how to attach a PC to the
drone, but basically it would be the same thing: just run the processing on a
PC.

Just by the title, I was expecting a clever way to run a pre-trained model
directly on a drone.

~~~
sarthakjain
Is there a title you would recommend I change it to, to make it more clear?

~~~
deckar01
Real Time Object Detection on Drone Video Stream

~~~
sarthakjain
Changed to "How To Do Real Time Object Detection On Drone Video Streams" in
the blog. Can't change it on HN.

------
MrTonyD
Wow. Combine this with the ability to drop some of the latest nano-sensors
(atoms strung together to build dust-sized sensors and batteries and power at
scale), and we should be able to monitor absolutely everything everywhere.
Let's see anyone even try to object to my rules or procure a weapon!

And given some on-board weapons, we would never have to worry about civil
unrest again. Only people grateful for the food and job they are allowed to be
in our society. Utopia!

~~~
sarthakjain
Hey MrTonyD. I understand where your coming from. But honestly most people are
using this for monitoring things like Solar Panels, Wind Turbines, detecting
seals and crop prediction. We can't have a sustainable future without a lot of
these technologies in place. [https://nanonets.com/drone/wind-
turbines/](https://nanonets.com/drone/wind-turbines/)
[https://nanonets.com/drone/solar-panels/](https://nanonets.com/drone/solar-
panels/)

All new technology will have some deviant use case associated with it. The
internet is one of the worst enablers of child pornography, human trafficking
and everything disgusting with the world. However it just means you try and
control the use of technology not throw the baby out with the bathwater.

~~~
MrTonyD
Well, I'm 60 now and I've worked in Silicon Valley for over 40 years. So I
guess I hear about all the other uses of such technology. I've worked with
NSA, CIA, DIA, all the branches of military, and a bunch of people who don't
officially exist. So, while there are definitely things I can't say, I will
say that the "dust sensors" have been in use for many years, and I'm not
describing anything theoretical. We need to think about these things in order
to address them. Simply saying that there also exist good uses isn't enough.

~~~
sarthakjain
Given your expertise over the many years how would you recommend we control
the use of such technology?

Maybe my question is based on the assumption that you agree: stopping
development of the technology in the first place is not the solution.
Hopefully this assumption is correct.

~~~
MrTonyD
There is no way it can be stopped. Most of the development is secret and much
further along than people believe. I think we need to regain community control
over those who possess this technology - it will be deployed whether we like
it or not. And, even more frightening, whoever deploys it first wins, and it
will be so effective that it can no longer be defeated. This issue is more
serious than global warming, jobs, healthcare, or World War 3 - it is about
having any freedoms beyond those provided by whoever deploys these monitoring
devices.

~~~
marvin
Infantry weapons powered by the kind of computer vision and facial recognition
AI easily demonstrable today would be incredibly frightening, no matter how
they were used. Of course there's no way of stopping this in an adversarial
world.

Surely it would only be a matter of choosing to increase the efforts for a
year or so in order to take these weapons from proof-of-concept to highly
effective. E.g. autonomous flying gun platforms that can be deployed en masse
and let loose with small or no human oversight (e.g. "hold the button to kill
the person the machine has identified as a valid target", if even that level
of control), or the kind of flying, target-seeking grenades envisioned in the
linked video. With any level of autonomy, loitering and communication with
command. It's only ethics and public perception that stops this today. In
total war, a 20% mis-classification rate doesn't matter. Bombing doctrine in
WW2 was much, much worse than this in its effect on civilians. Hell, many
military leaders emphasized killing civilians as a desirable objective. Even
present-day attacks consider "collateral damage" acceptable, even if
politicians sometimes pretend otherwise.

Defending against weapons like that would be an arms race in itself, and
completely beyond the reach of weaker actors. At least on a tactical level.
Maybe there's strategies that could be employed as a low-cost defense, such as
staying out of sensor range or confusing the sensors with adversarial
patterns, but it would crush the resistance of anyone that tried a
conventional defense.

Does the post-WW2 doctrine of emphasizing trade and the un-profitability of
war, or ensuring that it is more economical for everyone to support a stable,
free democratic economic system, still have any hope of preventing such
weapons from being used?

~~~
MrTonyD
I agree with you. But I think there is a bigger "weapon". When dust size
monitoring is pervasive, how can anyone acquire weapons or coordinate any
resistance? Dust is everywhere. And teams of nano-weapons can kill people
(this has been reported as research for well over a decade, so anyone could
reasonably assume that unreported research has gotten a lot further.) You
don't need armies when nobody can effectively fight you or even acquire or
build any weapon of any sort. So I don't worry as much about WW3 - since WW3
can be prevented with the pervasive monitoring technology on the near horizon.
(There may be a WW3, but we have bigger problems coming up.)

~~~
starbeast
When the map starts to reach the same size as the territory, don't you then
need a map for the map? Pervasive monitoring sounds like one hell of a dataset
to search through. Sure, you can use it to follow things you have already
identified, but just because you have recorded something, doesn't mean that
you know that you have recorded it.

~~~
MrTonyD
Some people wonder why Cloudera (Hadoop and now Spark) had such high stock
prices. But the real investors and the real clients were not public. Now, we
all know that these systems are limited both in data volume and ability to run
all the Machine Learning algorithms - but this problem is looking more and
more solvable - and there are many who are trying outside of the public eye.

------
jvdo
I tried this on my i7 macbook pro, but my CPU usage went to 700% and basically
i never got any moving image. Is this normal? What kind of computer you need
to at least run this kind of real time?

------
athiercelin
Using the MobileSDK ([https://developer.dji.com/mobile-
sdk/](https://developer.dji.com/mobile-sdk/)) you can build this straight on a
mobile app. With the OnboardSDK ([https://developer.dji.com/onboard-
sdk/](https://developer.dji.com/onboard-sdk/)) you can run it on the drone
directly.

------
bronco21016
Has anyone attempted to do anything like this for SAR (Search and Resuce)? I”m
not sure where one would gather a good dataset for training a model.

~~~
manneshiva
Here are a few more who have tried it or use deep learning for SAR:
[https://www.planckaero.com/maritimedrone/](https://www.planckaero.com/maritimedrone/)
[https://flytbase.com/flytstories/search-and-rescue-
drone/](https://flytbase.com/flytstories/search-and-rescue-drone/)
[https://news.developer.nvidia.com/autonomous-search-and-
resc...](https://news.developer.nvidia.com/autonomous-search-and-rescue-
drones-outperform-humans-at-navigating-forest-trails/)

Data for training such models for practical use are difficult to get. Often,
one has to train a model on a different but related dataset that contains your
object of interest and ensure that your model generalizes well. These models
are highly sensitive to the data that they are trained on, so it is best if
you can gather data which is closer to the actual environment (in which you
want to use the model).

------
sarthakjain
We at Nanonets have been building a lot of Machine Learning models for drone
based image analysis. A frequent request from our customers is how do they use
our models to do real time analysis. So we decided to create a how to on
building real time object detection using a DJI drone.

Also useful if you're just looking to get the video stream to your
laptop/computer.

~~~
jayant123
Awesome, was looking for something like this. Will definitely try it out. What
rate (fps) were you able to process the stream at?

~~~
bicubic
If you really want this at low latency, you can also take the hdmi output of
the dji controller and plug it into a capture card.

------
msadowski
Very interesting! I really liked the result. Would you mind if I linked to
this tutorial in WeeklyRobotics[1]?

[1] [https://weeklyrobotics.com/](https://weeklyrobotics.com/)

~~~
sarthakjain
sure

------
sigi45
Skynet ist thanking you ;)

------
tmkc13
Does this work on parrot drones also?

~~~
sarthakjain
Apps like `nanostream` allow you to forward the drone's live stream to any
RTMP server, so this should work for Parrot drones too

