A bit of a rambling summary of a book, and a big read for a newspaper, may be tl;dr. Points out a lot of concerns, including markets, autonomous vehicles and military systems, from unexpected interactions in digital ecosystems, and predatory algorithms. I’d heard of some of these before, but the whole is a bit scary. Any of you IT guys care to comment?
3 Likes
Its impossible to fully test any software that’s not pretty trivial. Add that to the unwillingness to take the time and spend the money required to do a fairly decent job of documenting and testing code and you end up with the crap we have now. Poorly understood, poorly tested and an accident waiting to happen in most cases. Add into that the speed of development and mind-bending complexity of anything with some form of self-learning and it scares the crap out of me…
1 Like
3 Likes
For me the big concern is the lack of willingness to be accountable. Algorithms are (originally, at least) created by people: some people come up with the desired principles and others code them. But nobody seems willing to own the output fully.
In some ways this is reasonable, because it’s really hard to test very complex systems. But that doesn’t mean you shouldn’t have real accountability.
Take the example of the cyclist killed by the autonomous car. The image recognition software didn’t work out what she was, as it was an unusual shape. It took some bad assumptions and then gave up, handing back control to a driver who had no time to react.
This is an interface issue. Sure the image system could be improved, but there needs to be a better way for the driving system to deal with uncertainty from the image system (brake if you’re not sure what something is?), and for uncertainty to be flagged to the driver. It’s just the kind of thing that everyone will say is someone else’s responsibility.
If there were more ownership of output, those owners would mandate much greater testing regimes, and have better ways for end users to feed back on their experience. There would have to be much more transparency. Unfortunately there’s too much money in it, so things remain closed due to commercial sensitivity.
Getting more and more human by the day then… 