Weekly Links: Glue, Org Charts, and Patreon’s Growth

August 16, 2017 • #

⚗️ Amazon Announces AWS Glue

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata

Interesting new service from AWS (is there a need in computing they don’t cover at this point?), providing serverless ETL transformations on datasets hosted anywhere. The automatic discovery is particularly interesting for applications dealing in highly variable data structures.

🏢 The Strategies and Tactics of Big

A conversation between Benedict Evans and Steven Sinofsky on big companies, their org charts, and what makes each (and their products) different.

💵 Inside Patreon

Patreon is still tiny compared to Kickstarter, where 13 million backers have funded 128,000 successful campaigns, but it’s rapidly growing. Half its patrons and creators joined in the past year, and it’s set to process $150 million in 2017, compared to $100 million total over the past three years.

This is a fascinating company, creating a funding mechanism for independent creators with a different model than the Kickstarter structure.

Weekly Links: Ambient Computers, Drones, and Focus

June 1, 2017 • #

💻 The Disappearing Computer

For his final weekly column of his long career, Walt Mossberg talks about what he calls “ambient computing”, the penetration of IoT, AR, VR, and computers throughout our lives:

I expect that one end result of all this work will be that the technology, the computer inside all these things, will fade into the background. In some cases, it may entirely disappear, waiting to be activated by a voice command, a person entering the room, a change in blood chemistry, a shift in temperature, a motion. Maybe even just a thought. Your whole home, office and car will be packed with these waiting computers and sensors. But they won’t be in your way, or perhaps even distinguishable as tech devices. This is ambient computing, the transformation of the environment all around us with intelligence and capabilities that don’t seem to be there at all.

🚁 Drones Go to Work

Great piece from Chris Anderson on the prospects of the commercial drone space. He makes great points about the true success of the technology being its penetration into business applications:

Although it might surprise you, I hope the future of drones is boring. As the CEO of a drone company, I obviously stand to gain from the rise of drones, but I don’t see that happening if we are focused on the excitement of drones. The sign of a successful technology is not that it thrills but that it becomes essential and accepted, fading into the wallpaper of modernity. Electricity was once a magic trick, but now it is assumed. The internet is going the same way. My end goal is for drones to be thought of as just another unsexy industrial tool, like agricultural machinery or generators on construction sites — as obviously useful as they are unremarkable.

Can Do vs. Must Do

Another good reminder from Fred Wilson on the importance of focus. He suggests setting no more than 3 “big efforts” in a year, the “must dos”. More than that is lying to yourself and losing steam on the ones you really care about:

But regardless of whether you have two, three, or four big efforts this year, you should test all of your initiatives agains the “must do” vs “can do” test. Just because you can do something doesn’t mean you should. I’ve written about the importance of strategy and saying no. Strategy isn’t saying no. It is figuring out what is the most important thing for your company and deciding to focus on it and say no to everything else.

Weekly Links: LiDAR, WannaCry, and OSM Imagery

May 18, 2017 • #

🗺 LiDAR Data for DC Available as an AWS Public Dataset

LiDAR point cloud data for Washington, DC, is available for anyone to use on Amazon Simple Storage Service (Amazon S3). This dataset, managed by the District of Columbia’s Office of the Chief Technology Officer (OCTO), with the direction of OCTO’s Geographic Information System (GIS) program, contains tiled point cloud data for the entire District along with associated metadata.

This is a great move by the District to make high value open data available.

🖥 WannaCry and the Power of Business Models

Ben Thompson breaks down the blame game of the latest zero-day attack on Windows systems. This article makes a great case for the business model being to blame rather than Microsoft, their customers, the government, or someone else. a SaaS business model naturally aligns incentives for everyone:

I am, of course, describing Software-as-a-service, and that category’s emergence, along with cloud computing generally (both easier to secure and with massive incentives to be secure), is the single biggest reason to be optimistic that WannaCry is the dying gasp of a bad business model (although it will take a very long time to get out of all the sunk costs and assumptions that fully-depreciated assets are “free”). In the long run, there is little reason for the typical enterprise or government to run any software locally, or store any files on individual devices. Everything should be located in a cloud, both files and apps, accessed through a browser that is continually updated, and paid for with a subscription. This puts the incentives in all the right places: users are paying for security and utility simultaneously, and vendors are motivated to earn it.

🛰 DigitalGlobe Satellite Imagery Launch for OpenStreetMap

DG is opening up access to imagery for tracing in OpenStreetMap, giving the project a powerful new resource for more basemap data. Especially cool for HOTOSM projects:

Over the past few months, we have been working with several of our partners that share the common goal of improving OpenStreetMap. To that end, they have generously funded the launch of a global imagery service powered by DigitalGlobe Maps API. This will open more data and imagery to aid OSM editing. OSM contributors will see a new DigitalGlobe imagery source, in addition to imagery provided by our partners, Bing and Mapbox.

📷 Updating Google Maps with Deep Learning

If you’re in the mapping space, seeing any of this R&D that Google is doing is mind-boggling.

Weekly Links: Podcast Edition

May 4, 2017 • #

🚗 The Man Behind Uber

The Daily is the New York Times’ daily radio show, which I’ve been enjoying lately. This episode is a companion to their recent piece on Travis Kalanick, Uber’s CEO.

🚢 Containers

Containers is an audio documentary on global trade and container shipping. Alexis Madrigal dives into the processes that bring things like coffee from a farm in Ethiopia to your local hipster coffee shop.

🚀 Nukes

The crew from Radiolab looks at the nuclear arsenal chain of command. At their invention, atomic weapons were treated like other military munitions: the military leadership had authority to use them like other conventional weapons. Over time we implemented the system we have now, requiring presidential authorization.

Weekly Links: Cartography's Future, Interactive Maps, and Building Moats

April 27, 2017 • #

🚙 Cartography in the Age of Autonomous Vehicles

An excellent, extremely detailed analysis from Justin O’Bierne on how maps and cartography might evolve if autonomous vehicles negate our need for turn-by-turn navigation.

We can’t apply today’s maps to tomorrow’s cars – but this is exactly what those who think cartography is dying are doing. (It’s not that we’ll no longer be navigating, it’s that we’ll be navigating different things – and we’ll need new kinds of maps to help us.)

🌎 Few Interact With Our Interactive Maps–What Can We Do About It?

Brian Timoney’s done some great writing on this topic over the last few years. In the GIS world, enormous amounts of money are spent by governments to build and host map portals. The goals are typically noble (transparency, openness, providing access to citizens), but the results are mixed. Much of the spend is in making the information interactive. The dirty secret is that people don’t actually interact with these maps. He proposes a number of ideas of how to get the best of both: lower costs to create with the same (or higher) consumer engagement. For example, static maps cost much less to create and could even do better at directing a reader to the right information:

Just because you’re publishing a map to the web, doesn’t mean it has to be a web map. If a user is only going to spend 10-15 seconds with your map without interacting, why spend two weeks wrestling with your Javascript? And the great thing is the focus a static map brings–a single view, a single story: don’t bury the lede.

💡 The New Moats

Jerry Chen from Greylock thinks “systems of intelligence” will be the next business model for software companies to create defensible value. He differentiates “systems of record” and “systems of engagement” as two layers in a stack of software applications that have existed since the dawn of the IT revolution in the 1990s.

These AI-driven systems of intelligence present a huge opportunity for new startups. Successful companies here can build a virtuous cycle of data because the more data you generate and train on with your product, the better your models become and the better your product becomes. Ultimately the product becomes tailored for each customer which creates another moat, high switching costs.

Aerial imagery with the Mavic

April 24, 2017 • #

At work we’ve been building an integration between Fulcrum and DroneDeploy, a service for automating drone flight and data capture for aerial imagery. It’s compatible with the Mavic, so I gave it a shot with some test flights over my house.

The idea is simple: use DroneDeploy to draw on a map the area you want to survey from above, and their app handles building the flight plan, sending it to the drone, and flying the waypoints to take all the photos. You then take the pictures from the drone’s storage and upload to your DroneDeploy project for processing. It stitches them into a single mosaic and does a few other data processing functions to give you maps of NDVI plant health, elevation, and even a 3D model of the scene.

Aerials of my house

This data is from a 3 minute flight over my house at about 150 feet. The post-processed scene reports 0.75 acres at 0.6 in/pixel resolution. Only 13 stills required to create this image. It’s pretty impressive for a few minutes of setup and a few minutes of flying. In the full-res images you can actually see Elyse and I clearly standing in the backyard. She was a little spooked as it took off, but loved the landing!

Weekly Links: OSM on AWS, Fulcrum Editor, & Real-time Drone Maps

April 21, 2017 • #

Querying OpenStreetMap with Amazon Athena 🗺

Using Amazon’s Athena service, you can now interactively query OpenStreetMap data right from an interactive console. No need to use the complicated OSM API, this is pure SQL. I’ve taken a stab at building out a replica OSM database before and it’s a beast. The dataset now clocks in at 56 GB zipped. This post from Seth Fitzsimmons gives a great overview of what you can do with it:

Working with “the planet” (as the data archives are referred to) can be unwieldy. Because it contains data spanning the entire world, the size of a single archive is on the order of 50 GB. The format is bespoke and extremely specific to OSM. The data is incredibly rich, interesting, and useful, but the size, format, and tooling can often make it very difficult to even start the process of asking complex questions.

Heavy users of OSM data typically download the raw data and import it into their own systems, tailored for their individual use cases, such as map rendering, driving directions, or general analysis. Now that OSM data is available in the Apache ORC format on Amazon S3, it’s possible to query the data using Athena without even downloading it.

Introducing the New Fulcrum Editor 🔺

Personal plug here, this is something that’s been in the works for months. We just launched Editor, the completely overhauled data editing toolset in Fulcrum. I can’t wait for the follow up post to explain the nuts and bolts of how this is put together. The power and flexibility is truly amazing.

Real-time Drone Mapping with FieldScanner 🚁

The team at DroneDeploy just launched the first live aerial imagery product for drones. Pilots can now fly imagery and get a live, processed, mosaicked result right on a tablet immediately when their mission is completed. This is truly next level stuff for the burgeoning drone market:

The poor connectivity and slow internet speeds that have long posed a challenge for mapping in remote areas don’t hamper Fieldscanner. Designed for use the fields, Fieldscanner can operate entirely offline, with no need for cellular or data coverage. Fieldscanner uses DroneDeploy’s existing automatic flight planning for DJI drones and adds local processing on the drone and mobile device to create a low-resolution Fieldscan as the drone is flying, instead of requiring you to process imagery into a map at a computer after the flight.

Mavic Pro First Impressions

April 19, 2017 • #

I bought a Mavic Pro a couple weeks ago and just got a chance to take my first flights this past weekend. In short, it’s the most impressive technology product I’ve used in years. I’ve never owned any drone, so this is pretty cool for someone in the mapping industry. Let’s dive in.

Mavic Pro

Since going out to fly aerial mapping missions with some partners of ours a couple months back, I wanted to buy one of DJI’s drones — either the larger Phantom 4 Pro, or the smaller Mavic. Extensive research led me to the portability and almost-equivalent technical specs of the Mavic over the P4. It’s so close in most of its capabilities, but the compactness of it is remarkable. I got the kit with the carrying bag, and it’s so small you could literally take it anywhere. I love the prospect of having this as a photography platform while traveling.

I did my first test flight in the backyard, plopped it down on the patio and kicked on the drone and remote control. Everything linked up right away and the DJI Go app was “Ready to Fly”. It’s so simple it seems like you’re doing something wrong. It feels like there should be more configuration. As long as you’ve got a clear GPS signal and you’re in “beginner” mode, you can just take off.

My first reaction was how easy it is to fly. You don’t have to do anything and the drone just hovers. Let go of the controls at any time and it stays put. The controller sensitivity feels smooth and intuitive; I was strafing sideways, rotating, and descending to create cool sweeping shots within 2 minutes. With a little practice you could do pro-level photography with this. Landing was just as easy: you descend where you want to land and as you approach the ground the drone halts at about 18” using its collision detection sensors. With another long hold on the left stick, it initiates the landing sequence and slowly touches down. I also tried the “Return to Home” feature, which is enabled as long as you let the drone get a good locked home location before takeoff. It’s so cool to see it work. The drone can be away from you and when you tap Return to Home on the app, the drone comes home and makes a smooth and careful landing. In a couple of tests it came home and landed in a 5-10 foot radius from the takeoff point.

Next is the software. The DJI Go app is what you use when you dock your device with the controller to get the live video, heads-up display, and settings controls, and it’s an amazing piece of software. I hadn’t used earlier versions, but in version 4, you can control everything from the app. The video feed from the drone and the HUD view of all the needed metrics looks great (altitude, bearing, distance). Triggers on the sides of the remote snap photos and start recording video. DJI has honed the system down to the simplicity of a video game. I’ve only done a couple of flights, but the video and photo quality is excellent. 4K video from this tiny airframe and camera is a stunning feat.

One of my flights was in about 15 knot winds, and the little guy held up well. The camera’s gimbal was rock steady even in breezy conditions. I noticed a tiny bit of jitter when flying into the teeth of the wind, but not enough to make a difference. I flew one mission of aerial imagery with DroneDeploy, but will dive deeper on that in a future post when I can do more flights.

A few other things on the docket to try:

  • Object detection and tracking — you can lock onto a moving object and the drone and camera will follow. When I find a use case for it I’ll try it out and report back. Looks neat from videos I’ve seen.
  • Flying at high altitude — so far I haven’t gone above about 150 feet.
  • Flying at longer ranges — haven’t yet gone farther than a few hundred yards away, but the range on this thing is huge. When I get more confident with it I’d like to do some longer flights for cool video. Thinking about our Florida Keys trip to Marathon in June!

Weekly Links: Tensor Processing, Amazon, and Preventing Traffic Jams

April 13, 2017 • #

Google’s “Tensor Processing Unit” 💻

Google has built their own custom silicon dedicated to AI processing. The power efficiency gains with these dedicated chips is estimated to have saved them from building a dozen new datacenters.

But about six years ago, as the company embraced a new form of voice recognition on Android phones, its engineers worried that this network wasn’t nearly big enough. If each of the world’s Android phones used the new Google voice search for just three minutes a day, these engineers realized, the company would need twice as many data centers.

Jeff Bezos’ Annual Letter to Shareholders 📃

An excellent read. Their philosophy of experimentation comes through. I liked this bit, on the “velocity” of decision making:

Day 2 companies make high-quality decisions, but they make high-quality decisions slowly. To keep the energy and dynamism of Day 1, you have to somehow make high-quality, high-velocity decisions. Easy for start-ups and very challenging for large organizations. The senior team at Amazon is determined to keep our decision-making velocity high. Speed matters in business – plus a high-velocity decision making environment is more fun too. We don’t know all the answers, but here are some thoughts.

First, never use a one-size-fits-all decision-making process. Many decisions are reversible, two-way doors. Those decisions can use a light-weight process. For those, so what if you’re wrong? I wrote about this in more detail in last year’s letter.

Second, most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you’re probably being slow. Plus, either way, you need to be good at quickly recognizing and correcting bad decisions. If you’re good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure.

How not to create traffic jams, pollution and urban sprawl 🚘

The Economist analyzes the state of parking economics. The gist: free or low-cost parking equals congestion and more drivers roaming for longer. Some great statistics in this piece:

As San Francisco’s infuriated drivers cruise around, they crowd the roads and pollute the air. This is a widespread hidden cost of under-priced street parking. Mr. Shoup has estimated that cruising for spaces in Westwood village, in Los Angeles, amounts to 950,000 excess vehicle miles travelled per year. Westwood is tiny, with only 470 metered spaces.

Weekly Links: Cars, AI Doctors, and the Mac Pro's Future

April 6, 2017 • #

Cars and Second Order Consequences 🚙

The cascading effect of a world with no human drivers is my favorite “what if” to consider with the boom of electric, autonomous car development. Benedict Evans has a great analysis postulating several tangential effects:

However, it’s also useful, and perhaps more challenging, to think about the second and third order consequences of these two technology changes. Moving to electric means much more than replacing the gas tank with a battery, and moving to autonomy means much more than ending accidents. Quite what those consequences would be is much harder to predict: as the saying goes, it was easy to predict mass car ownership but hard to predict Walmart, and the broader consequences of the move to electric and autonomy will come in some very widely-spread industries, in complex interlocked ways.

A.I. versus M.D. 💊

Siddhartha Mukherjee looks at the potential for AI in medicine, specifically as a diagnostic tool. Combine processing and machine learning with sensors everywhere, and things get interesting:

Thrun blithely envisages a world in which we’re constantly under diagnostic surveillance. Our cell phones would analyze shifting speech patterns to diagnose Alzheimer’s. A steering wheel would pick up incipient Parkinson’s through small hesitations and tremors. A bathtub would perform sequential scans as you bathe, via harmless ultrasound or magnetic resonance, to determine whether there’s a new mass in an ovary that requires investigation. Big Data would watch, record, and evaluate you: we would shuttle from the grasp of one algorithm to the next. To enter Thrun’s world of bathtubs and steering wheels is to enter a hall of diagnostic mirrors, each urging more tests.

This piece is one of the best explanations of neural networks I’ve read.

The Mac Pro Lives

If you follow the Apple universe, you’ve surely heard the frustration of professional Mac users who’ve felt abandoned by Apple neglecting their pro hardware for 3 years. They’re resurrecting the lineup now with a redesigned Mac Pro. The craziest bit about this story is that Apple is coming out of the shell to talk about a new product months before launch, to a handful of select journalists.