Winnipeg Free Press - ONLINE EDITION

‘Crash’ course in Big-Data design

  • Print

Over the past two weeks we have seen the following computer system crashes:

• a three-hour network shutdown on Aug. 22 that paralysed the NASDAQ stock exchange, crippled others, and caused a one-third drop in the daily total of shares traded on American exchanges;

• also on Aug. 22, a blackout of Apple’s iCloud that lasted for 11 hours for some customers;

• a trading glitch in the Goldman Sachs computer on Aug. 20 that resulted in a large number of erroneous stock and options trades and cost the firm up to $100 million;

• a shutdown of Amazon’s North American retail site on Aug. 19 that lasted almost an hour and resulted in an estimated $2 million in lost sales;

• on Aug. 16, a four-minute global outage of Google’s services, including email, YouTube and its core search engine, that led to an 40 per cent drop in global internet traffic;

And last month, in another part of the forest, we had the director of the U.S. National Security Agency, Gen. Keith Alexander, admitting that he still did not know exactly which files whistle-blower Edward Snowden had downloaded and taken with him when he fled the country two months before.

Well, Gen. Alexander didn’t exactly admit it; he just declined to say whether he knew, but that comes to the same thing. Two months after Snowden flew the coop, the NSA still doesn’t know how many more of their embarrassing secrets are out there waiting to be revealed.

This may explain something quite puzzling that happened last week. A Brazilian citizen, David Miranda was changing planes in London when he was stopped by British police under the Terrorism Act, questioned for nine hours, and then released — but the police kept his computer, two pen drives, an external hard drive, and various other electronic items.

Miranda is the partner of Guardian journalist Glenn Greenwald, who has been working on Snowden’s documents, but the police wouldn’t have gone to all that trouble just to harass him, particularly since their actions were probably illegal: all their questions were about Snowden and the NSA files, not about terrorism. And why would they even bother to confiscate Miranda’s electronics? Don’t they realize there are bound to be copies elsewhere?

It’s less puzzling if you assume that the NSA asked for the operation (of course it did), and that its goal was actually to find out just how much Snowden knows, and can prove. Maybe it found out, maybe it didn’t — but what it tells the rest of us is that the NSA is not really in control of its own data. If Snowden can take it away with him, so can others.

There are 850,000 potential "others" — Americans with top secret clearance and access to the data — and some of them will not have the same high motives as Snowden for stealing the data. In fact, the NSA even catches an average of one employee a year who has been using the system to track a lover or spouse they suspect is straying. God knows how many it doesn’t catch – but if its inability to figure out what Snowden took is any guide, probably a lot.

What the NSA has built is a system that is too big to monitor properly, let alone fully control. The system’s official purposes are bad enough, but it cannot even know the full range of illegitimate private actions that it permits. And this is not a design flaw. It is inherent in the very size of the system and the number of people who have access to it.

Which brings us back to NASDAQ, Apple, Goldman Sachs et. al.

If it can be done, it will be done. Algorithms will be written for automated trading at speeds measured in fractions of a microsecond, and the competition will have to follow suit. It will become possible to store immense amounts of data in a virtual "cloud", and the cloud will take shape. It will become theoretically possible to listen in on every conversation in the world, and the surveillance systems to do it will be built.

Every step onward increases the scale and complexity of the systems, until they are too big and complex for any one person to understand. They will run without supervision, for the most part, and when they fail (as they must from time to time) the failure will also be hard to understand. And if you give hundreds of thousands of people access to the system, your secrets will not stay secret for long.

The volume of date moving on the internet and private networks is expanding very fast at the moment — from six gigabytes for each person on the planet this year to 16 gigabytes per person per year by 2017 — and system design is not keeping up. Given time, it may be possible to catch up on that front, if the rate of expansion eventually slows. But it will be much harder, maybe impossible, to build leak-proof surveillance systems.

 

Gwynne Dyer is an independent journalist whose articles are published in 45 countries.

 

Fact Check

Fact Check

Have you found an error, or know of something we’ve missed in one of our stories?
Please use the form below and let us know.

* Required
  • Please post the headline of the story or the title of the video with the error.

  • Please post exactly what was wrong with the story.

  • Please indicate your source for the correct information.

  • Yes

    No

  • This will only be used to contact you if we have a question about your submission, it will not be used to identify you or be published.

  • Cancel

Having problems with the form?

Contact Us Directly
  • Print

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

Have Your Say

New to commenting? Check out our Frequently Asked Questions.

Have Your Say

Comments are open to Winnipeg Free Press print or e-edition subscribers only. why?

Have Your Say

Comments are open to Winnipeg Free Press Subscribers only. why?

The Winnipeg Free Press does not necessarily endorse any of the views posted. By submitting your comment, you agree to our Terms and Conditions. These terms were revised effective April 16, 2010.

letters

Make text: Larger | Smaller

LATEST VIDEO

Keri Latimer looks for beauty in the dark and the spaces between the notes

View more like this

Photo Store Gallery

  • A gosling stares near water at Omands Creek Park-See Bryksa 30 day goose challenge- Day 25– June 21, 2012   (JOE BRYKSA / WINNIPEG FREE PRESS)
  • Ruth Bonneville Winnipeg Free Press January 18, 2011 Local Standup -

View More Gallery Photos

Poll

Should the city grant mosquito buffer zones for medical reasons only?

View Results

View Related Story

Ads by Google