#041 – Bringing Advanced Computer Vision To Industrial Drones with Alex Harmsen

welcome to commercial drones fm the podcast that explores the commercial drone industry the people who power it and the concepts that drive it I'm your host Ian Smith hey everybody before we get started I want to let everyone know that we hit our first patreon funding goal so this means everyone from here on out that donates $1 a month to the podcast gets an exclusive invite to the private patron only slack group that I created to discuss everything commercial drones so you can get your invitation to the group by visiting patreon.com slash drones podcast that's p80 ar e n comm slash drones podcast and just by donating a buck a month to the podcast you're in ok so welcome to commercial drones FM now that that's over with thanks everybody for listening and tuning in so this evening we're sitting in San Francisco at the drone deploy offices and I am here with Alex Harmsen who is the CEO and co-founder of iris automation which is a drone startup out of Y Combinator that recently raised one and a half million bucks in seed funding to bring sense and avoid technology to drones used for industry so thank you so much for coming and being here with us this evening Alex glad to be here again appreciate it man so let's get started let's let's kick this off with a little bit of backstory so we met actually previously at an orange Silicon Valley event orange as Americans would say like the telecom company and they have like a venture capital arm that they have here in Silicon Valley and you were speaking on a panel on it was like artificial intelligence in drones or something yeah I think so it was artificial intelligence machine learning in general how drones can be smarter or how that's changing over the next few years nice yeah so we met at the little you know cocktails afterwards in typical French fashion there was some cheese and wine it was really nice so at that point after hearing you on the panel I knew I wanted to get you guys on and at that time you had not raised a seed round as far as I'm aware and and you have some big news now you guys were were all over in in the news for getting your seed funding of one and a half million bucks so congratulations on that good thank you very much yeah no problem so tell us a little bit about first of all what is iris Automation I mean the kind of tagline sense an avoid technology for drones in the industry and I saw one of your investors compared you guys to airbag and seatbelt manufacturers from an earlier generation so your technology is kind of similar to that and I guess that means safety I mean a big part of it is trust I think in general like what we've seen in in consumer drones and military drones has been this huge uptake but like that same sort of uptake hasn't quite happened on the commercial or industrial side and I think a big part of that is its trust and safety I think in general like comparing it to airbags is like quite a great idea just because like airbags are so prevalent and you're not gonna find any car today without airbags I mean we see that as the future of drones as well especially autonomous be online a site may every drone should have some sort of sense and avoid system on board that's really at iris automation rebuilding computer vision bastes collision avoidance systems specifically targeted for industrial drones okay cool can you explain what exactly like what specifically makes an industrial and industrial drone a part of it is size part of it is cost but really the the big the big component there is with what it's being used for and so these are drones that are being used for forestry management and agricultural surveys mining exploration package delivery human transportation even biplane inspection anything like this I mean we're covering larger distances we want these drones to be completely autonomous and scale becomes a big part of that and so what led you to create the company and how did it happen your co-founder so there's another founder involved as well right James Howard and so we had met years back in University and Vancouver and British Columbia and we had started up this drone team together and grew it from the two of us and a third guy Chris 247 engineers and we were competing in national competitions we drove across Canada I mean this was very exciting we're the first thing to ever fly multiple drones in the air at the same time we had a computer vision component to that and I mean it was sort of like a small company I mean it's 47 engineers volunteer basis and we built I think eight different drones over those years and I mean he went off and worked at spire and Boeing and I worked at Madden at NASA and at a certain point I mean we always will is just stayed in touch and and when we saw this growth in the drone industry and we were both really excited about aerospace in general and robotics and essentially saw this huge divide and this artificial constraint on the commercial drone market and said we should try and figure out how to solve this trust issue and like ultimately that's what created iris automation a certain point we quit and moved back up to Vancouver and worked in a basement for a while writing software and then said you know what we have the prototype and let's let's get some investment and pour some gas on this fire that's awesome so your base here now in the Bay Area exactly so we just moved down so we came down for Y Combinator over the summer and I mean so much such a great ecosystem here there's talent investors just in general these sort of events great podcasts to be a guest on a podcast and also to get investment for a drone company were there other so for those who don't know what Y Combinator is can you explain and what that is exactly sure it's a one of the bests no and accelerator programs in the world and essentially they select the number of different companies twice a year in two different batches and they're brought into Mountain View or San Francisco and essentially they bring in mentors guests there's something called demo day at the very end where they put you on a stage for two and a half minutes in front of 500 investor earth was quite quite thrilling probably the most rehearse two and a half minutes of my life but I mean the sort of interest that came after that was phenomenal oh that's awesome and how long oh so that goes on for like four three four or six months or something and y combinator yeah it was three months Jim – end of August I think that the biggest part of it is being part of this alumni network now and so there's something like a thousand founders that have gone through that program and just being part of that network I mean being sort of on that inside circle has been tremendously useful for for contacts even just asking questions I mean there's a forum that I could pose any question to and it could be CEO of X Y Z that company or it could be some other founder of a much smaller company that made mistakes eight years ago and I was wanting to share those experiences that's a huge asset um was being a drone company unique at that time like for Y Combinator where you guys like one of the only the few in there or one of the only ones I actually don't know if I'd consider myself a drone company ooh I mean really we're going after that computer vision the artificial intelligence for these autonomous vehicles and I mean I make so much sense to apply that the drone to make so much sense to focus on the drone industry but it's it's more providing that so far that intelligence for this industrial drones to allow them to do their thing rather than like being a drone company nice good answer okay cool so are humans bad at piloting drones them like can we not trust humans to safely pilot drones is that why IRS automation and computer vision all this all this technology for sentient avoid is that why it's needed or their alternative reasons I honestly I think humans aren't just bad at piloting drones humans are bad at piloting manned aircraft – that's true there's so many accidents and like most national and Transport Safety Board investigations essentially call into question human operator competence pilot error pilot errors the number one cause of any aviation accident anywhere and like drones are an extension of that the really the big issue with drones being operated is like once we essentially we want to be able to move to fully autonomous drones for a lot of these applications it makes sense it's so much more cost-effective and like these drums are essentially robotic I mean one of the main reasons why we have these drones is to be able to like extend this human reach or to have eyes in the sky or from a different perspective and it's it's not really natural for the humans actually be operating them it's counterintuitive to actually steer them with two little sticks with buttons I mean ideally these drones should just be able to accomplish their tasks without any sort of human interference and so something like iris automation being able to provide situational awareness I mean being able to letting these drones see the world the way a pilot would I mean that essentially gives us the confidence that they would fly those missions without running into main aircraft or birds or balloons or buildings or power phone towers yeah whatever you have you okay cool so what so how many employees are you guys at today at IRS automation we're at ten people ten people cool and so what is the iris automation team like I mean are you guys I'm assuming mostly software developers at this time but right yeah how is the team comprised and what are you guys kind of working on like currently I mean it's it's quite a hard challenge right I mean we're talking about outdoor computer vision at a large range in real time on embedded systems and it's I mean the computer vision and deep learning itself is like no trivial challenge that's attractive though to software that's super attractive I think we've got most of the buzzwords covered which is just crazy IM LCV there you go and so I mean all of the team as engineers except for one okay and so we've we've really like made that team engineer heavy just because like we know we're trying to solve a hard problem and we want to make sure we solve the problem and make the most reliable product before we really start selling something mmm nice and are you guys looking for other people to join your team I like to always ask this question because some people might be listening who who Mari maybe are curious about joining the commercial drone industry or we won't call it the drone industry but for the sake of this oh we will we won't call you guys a drone company right now but yeah what's the deal you guys hiring stuff what are you doing with the one and a half million bucks maybe that's a better question sure I'm in a big part of that is hiring and partly partly it's I mean keeping that current staphon and i mean moving us from like our current product which is like in the field with pilot pilot customers to something which is actually commercialized like something commercializable and something we can sell in a much more widespread like official version and the other part is I mean we're hiring a testing or reliability engineer and we're always looking for more computer vision people because I mean ultimately this is where the real magic happens speaking of computer vision so how how are you guys implementing computer vision machine learning and AI kind of all together into the drones I mean like if you can just describe kind of maybe what's happening behind the scenes briefly without of course revealing any trade secrets and whatnot but yeah how would you describe that in an a medium which is only sound so people can only listen to this is a good luck I mean really we've taken a layered approach to that and we're really we we have a number of different deterministic like pieces to what we're doing and like one of the fall backs of computer vision sort of traditionally has been it fails catastrophically or machine learning or artificial intelligence in general could feel catastrophic ly to go from like 95% confidence to zero and so there's a certain certain elements of the system that just can't fail in that way and I mean this is very important for a safety product for something that has a big reliability element and so we've sort of mixed these newer tradition these newer AI deep learning algorithms for classification for tracking with these more traditional more deterministic algorithms that are able to do the sort of 3d reconstruction sort of tracking and seeing recognition that we actually want out of some sort of safety product and so that's I mean that's that's been very important for us to be able to certify a product like this fresh first I actually built something that we can stand behind and say this is a safe product this would actually help the drone move forward and so I mean part part of it as situational awareness we know a number of different elements about the scene part of it is dynamic tracking of objects and so we're looking at moving objects like birds helicopters aircraft at a range of over 500 meters and trying to figure out where they're gonna go so we're doing robust trajectory planning there and then so the other element that the system is being able to do 3d reconstruction at a huge range as well as hundreds of meters of range because I mean ultimately we feel like the best solution to this is complete situational awareness and so we want to know where is everything that's not moving and where is everything that is actually moving propagate that forward in time and if we have some sort of collision happening there then we want to notify the autopilot or the operator and how is this actually being implemented I mean is this just software onboard or is it is there a hardware component to this I mean really the magic happens in the software and so really I mean we see ourselves as a software a I company but I mean essentially you need a camera a single camera that takes in this video feed just like any human pilot and that happens I mean essentially that's transferred to a processor on board and essentially that processors doing all this computation it's running these algorithms and then relaying that back to the autopilot and so I mean for us we could sell software licenses for now we're also selling the hardware just because we we want some computing platform that we know of but I mean being able to sell reference hardware or being able to sell the hardware just at cost just make it easier to install I was like more more than willing to do that in the future maybe like API or SDK kind of compatibility are you guys envisioning that at all yeah well yeah I mean we have an iris API and then we have a couple like specific hooks for different auto pilots on the market oh that's awesome and I have to say are you guys compatible with DJI drones at the moment because I mean they're there the gorilla in the room yeah we are oh cool okay nice that's awesome so what what kind of compatibility does this have I mean okay you did mention like the vision component like you get a video feed and then it's analyzed by the software which is powered by hardware powerful hardware do you have any other sensors involved like I'm talking radar or any type of sonar or any anything that's you know using any other technology no really it's purely vision based I mean ideally we use other sensors and like I would love to do sensor fusion the issue is that like there's nothing on the market that can sends the sort of range that we want to be able to sense that or is like anywhere remotely in the price range that we want to be able to sell the system at and so I mean hopefully radar and lidar systems like over time in the next four or five years come down in price come down in weight volume like become more energy efficient and we could integrate those into the system but for now it looks like computer vision is sort of the only solution that would work in this case makes sense yeah tiny little cameras they can see really far they don't weigh a lot they have very inexpensive makes a lot of sense and the great thing about software is that like we could do a lot of software testing and very quick iterations on this sort of software without doing a lot of hardware or testing that is required for radar lidar system yeah yeah especially yeah every little ounce every little gram matters of course in the drone and a small drone world right now and what about like are you guys gonna integrate as far as with like air traffic control maybe I mean you know if are you gonna try two or maybe maybe pull in sources of other sources of information that can aid your technology in detecting and avoiding other aircraft or things in the sky or maybe just in the general vicinity all right I mean the an in aviation in general and more robust robotic systems like we're always going to get a better solution if we add in different sensors if we add in different sources of information so I mean like we're completely open to that we're talking with a couple different companies about butt fusing that information into some sort of grid I mean we're part of NASA UTM we're just trying to create this traffic management system and so to feed into NASA at UTM would be I mean it would be wonderful and it's definitely in our roadmap yeah bi-directional sync would be great and as far as miss tonight now is 500 meters or so or guys okay cool and that's all just based on the vision and stuff so if the camera if the sensor improves like if you have more pixels basically does that mean better performance necessarily is there like a exactly the same oh cool like honestly we could throw a very powerful camera on there and we would be able to see a lot further part of it is like computational power constraints which also come down over time that's right right I mean as as we get more I mean better and better processors the system just naturally improves and as we get better cameras the system naturally improves I mean that's there's some trade-off there right now and we're not even using the most expensive or highest resolution camera in the market and we really want this to be the software and I mean naturally over time like other factors or smartphones or processing power comes down to cost or increases in efficiency and essentially we just get the right those those waves yeah and what kind of like if you look so there's already some computer vision sensing you know products on the market obviously there's DJI they have their vision sensing also intel has real sense how do you compare to those products like how does iris automation differentiate itself I guess maybe even from like the industrial aspect like what are you guys doing that tackles the problem or solves the problem differently than those two I mean that's a that's a great question the question like that comes up all the time and it's I think for me relatively simple to deal with but it could be it could be hard to just intuitively understand I think when it comes to to collision avoidance in general or different colors collision avoidance technologies we're sort of broken it up into two categories whereas intel realsense the DJI phantom4 with their collision avoidance falls into this short a short-range bumper solution this is 1020 meters in front of the drone it could be multiple directions but it's really meant for consumer drones it's meant for a sort of very slow short-range navigation it's it's meant to fly through a parking garage or through a forest or it's special follow-me features it's more interaction with humans or if you're using it for wedding shoot it just it's there to make sure you don't hit the altar accidentally but then the other side of that the other side of collision avoidance is more the sense and avoid technologies and that's more what we're going after this is more full range solutions this is 10 to 500 meter range it's I mean the honestly the point of our system is to make drones boring mmm I mean we want to make boring a safe an aviation that's that's what you want to go for I mean it's like certain orders of magnitude higher in safety we want to make sure that these drones aren't running into manned aircraft I mean we want to be able to allow these drones to fly in national airspace and for the FAA and Transport Canada and whatever other regulatory agencies that feel safe about letting hundreds of thousands of package delivery drones like loose in airspace yeah and that that's the sort of problem we're dealing with that's the sort of problem that we're solving with this collision avoidance technology and I mean that in my mind that's a huge differentiator it's more toys compared to aviation and avionics earlier and that's a good point because earlier you had mentioned and I'm probably going to butcher this trajectory computation or something basically trajectory propagation what is that exactly so of what I imagine is okay you're the camera see something your software identifies there's an object and that it's and then it tries to figure out where it's going exactly the trajectory yeah so I mean we can recognize what sort of aircraft we're looking at or if it's a bird or helicopter or plane and I mean essentially we have a number of different trajectory models there we're looking at like how it may fly through the sky I think similar to a pilot I mean if you were a pilot in an aircraft and a little Cessna and you saw another plane or if you saw helicopters ipping along some landscape you could figure out where it's gonna go in the next 20-30 seconds and we're doing the same thing so we're looking at where it is where it's going trying to figure out what it is and I mean ultimately as our algorithms get better we get better and better at this trajectory propagation and essentially the more reliably we can project that into the future the more likely we'd be able to detect some sort of collision and actually get out of the way okay so you'll send a command to the aircraft through the system and it will just move and alter trajectory if it looks like it's gonna be a collision exactly and that's completely configurable by the user I mean some users want to pull the parachute immediately some just want to land immediately some just one hover some actually want to move out of the way and add a few new GPS waypoints into that mission do you think there could be you know once this technology becomes more widely adopted and maybe you know UTM is in place and everything do you think there could be regulations on how far a autonomous vehicle would have to take evasive maneuvers from another object if using a specific trajectory propagation technique determines that there's an x percent chance you know you see where I'm getting at like definitely and we're talking with regulators right now about that I mean we're involved and we were on subcommittees with our TCA and ASTM and NASA UTM were involved in like drone representation groups in Canada of unmanned systems Canada I mean a big part of that trying to figure out how do we integrate drones into the National Airspace how do we integrate drones and have them fly in the same space as manned aircraft is setting these basic rules and I fully expect there to be regulations like that and like every government and every regulatory body we've talked to have said that you need to mitigate the risk of collision with manned aircraft or they've said you specifically need sense and avoid technology to be able to do that the big question right now is what is good enough what is safe enough and I mean as a thought leader in the space we're definitely pioneering those concepts and proposing certain solutions for that I mean it's not something that we're just pulling out of the air this is we've done a lot of simulations there we've built a simulator from scratch that allows us to be able to do these tests and allows us to do not just a couple drone flights but hundreds of thousands of hours of simulations in real world environments and so it's it's that sort of robustness or it's that sort of thinking that actually allows us to build certification procedures it allows us to to actually say this is good enough because like as much as we want in aviation at a certain point we have to draw the line and say like this many accidents are acceptable and I mean we want to reduce that as much as we possibly can and we feel like if you add an iris automation system onboard iock suddenly a drone becomes x times better x times safer and we know that's gonna help in selling to mining companies or oil and gas companies or forestry or even helps Amazon get their package of delivery drones out actually flying in the u.s. so different companies in different industries assess risk differently there are more you know higher or lower tolerances of risk now you've got me thinking what if the vision systems trying to detect things not to run into we're not just pointing straight ahead but also pointing down like right now you need a special waiver from the FAA in the u.s. to fly over people could this technology theoretically be like pointing down and then seeing people underneath it and then you know assessing risk that way and keeping itself at a safe distance to not like if there was an engine failure or hit the people or something sure I mean like really what we're doing is building that software that that intelligence that allows us to do that complete situational awareness and we're pointing up down forward backwards left-right doesn't really matter and then the current implementation of the system is a camera with a certain field of view facing forward because most drone companies and most regulators are worried about hitting something head-on or colliding with something in sort of that this essentially going forward limits that time to collision and so you that's the direction where you really want to be looking at but I mean definitely there's no real limitation for us they're just add more cameras or like maybe certain drone companies are worried more about stuff hitting them from behind or from the bottom and you should just point the system in that direction I just had a paranoid bigbrother moment where the camera is actually pointing back at the operator and then if the operator loses the camera loses sight of the operator comes back home because you can't do visual line of sight or beyond visual line of sight that would be just really Kurt yeah seeing how this technology matures and then seeing regulations get bundled in with it like oh if you're doing this if you're making this type of system deployed on a commercial unmanned aircraft system it must be able to come you know ensure compliance with XYZ regulations that's really interesting it's pretty exciting I mean we're with part 107 and sort of like the end of 33 exemptions there's a whole new set of roles but at the same time in this especially in industrial drones or commercial drones beyond visual line of sight is sort of the Wild West it's sort of we don't really know how that's going to develop and like we want to be at the forefront of that or we want to be an enabling technology in that element of the drone space and make those rules or help make those rules or suggest those rules and essentially it's like see where that takes us yeah technology's gonna play such a huge factor in that of course in computer vision as it permeates the drone industry further so I have to ask then mr. Harmsen what is your favorite drone of all time and why what a great question so before iris Automation I was working at NASA at the Jet Propulsion lab and we were working on a project called Mars helicopter whoa and essentially this was a drone for the Martian surface to help guide the Mars rover along the Martian surface I mean this was very interesting we were working on computer vision systems specifically for that drone to be able to do 3d reconstruction of that surface but I mean that the Mars rover has the same issues that like industrial drones have now honestly it's 2.4 million or 4 billion dollars and we just don't trust it to navigate on its own as soon as it goes beyond line of sight as soon as it actually lands on the Martian surface we're worried about it falling off a ledge as high as this table flipping over and boom like people like hey so there's like a certain situational awareness like on the Mars rover but I mean having some sort of drone that follows it along and sort of gives it ice in the sky to map out that landscape I mean that's that's pretty exciting and has it come on yet and that's a very long project and we will see if it actually comes to fruition but no doubt that that's my favorite drill I mean that's an excellent answer what the helicopter so its uses the same I mean it uses air molecules to stay aloft yeah Oh cuz that okay the gravity is less and it can okay what exactly little pressure but less gravity as well okay well geez great answer like DJI phantom 3 I'm gonna go with Mars helicopter drone so that's a that's a really great answer now what is the future hold then for iris automation I mean looking forward you know a few years in the future where are you guys headed where do you want to be what are your aspirations I mean I'm sure this came up and all the you know pitches and stuff for funding and everything but you know where do you want you know if you look at the company in a few years and the way the industry is headed like where what's your ideal future look like for iris automation yeah I never get tired of talking about that and answering these sorts of questions I mean this is I mean sense and avoid technology is exciting and like safety systems in general for these drones are extremely important and this is really what's gonna get us that next level unlock being beyond visual line of sight flights unlock fully autonomous drones but like once that is there I mean that's really when stuff gets interesting and stuff gets crazy on there on this commercial drone side I mean there's so many applications for drones and the I mean all these applications that live applications I listed before from human transportations a force or a management or like even forest firefighting and the great thing about robotic systems like this is that like right now they're relatively dumped I mean these auto pilots are I mean they're essentially just following GPS waypoints through the sky but like wouldn't be cool if instead of just plotting GPS waypoints to follow a pipeline you just told it like go and follow the pipeline and find me where the leaks are or if you said like hey like send out a couple drones and said there's a forest fire somewhere here because there's smoke in the air go and find that map it and then just autonomously put out the fire cuz we have a couple of these drones that are just water bombers and just filling up from lakes autonomously and this is this is definitely like in a realm of science fiction but also like five ten years away that's very good I mean like if we can build the artificial intelligence if we can build the situational awareness for these sorts of like this new generation of autonomous drones I'm very happy to be behind that I like the future that you paint thank you so much so everybody you can check out iris automations website at iris on board calm that's IR is onvo ard calm and follow them on twitter at iris underscore and automation iris underscore automation and you can follow Alex mr. Alex Harmsen on Twitter at Alex harm that's al e x h a RM and of course while you're at it you can go ahead and follow the podcast on twitter at drones podcast and check out the brand spankin new website at commercial drones dot FM in your web browser it's a much better experience I spent a lot of time revamping this thing so I'm gonna focus on putting a lot more content on there and that's really all we have today of course you can check out the patreon page if you want to support the podcast at patreon.com slash drones podcast I'd really appreciate that any last parting thoughts before we go ahead and cut off the mics Alex I'm just happy to be here thanks for hosting me today and it's exciting to talk about the future of drones and I'm an AI I like to think that were onto something big here but it's also just the start of an industry and my everyone talks about all the use cases and all the good that drones will do and like hopefully we can be part of creating them totally agree well I'm very happy to be here too so again thank you Alex Harmsen everybody keep an eye on iris Automation see what they come up with next as we get towards a much smarter drone future we're gonna go ahead and cut off the mics thank you everybody for listening fly safe out there Cheers

Iris Automation is a Y Combinator startup who recently raised $1.5 million in seed funding to bring sense and avoid technology to drones used for industry. Buzz words like computer vision, deep learning, and artificial intelligence only begin to scratch the surface in describing what the company is building. To sense issues and avoid them, powerful onboard drone hardware allows Iris Automation’s software to track birds and other aircraft, predict their trajectories, reconstruct the scene in 3D at hundreds of meters of range, and then notify the drone operator of issues in real-time, or even take evasive action autonomously. Alex Harmsen is CEO and co-founder of Iris Automation and joins Ian to discuss why sense and avoid technology is a requirement for building trust in industrial drones and ultimately make them boring.

-Video Upload powered by

Leave a Reply

Your email address will not be published. Required fields are marked *