home

NSA Electronic Data Surveillance: Doomed From the Start

The Baltimore Sun has an expose of the failure of the National Security Agency's trailblazer program.

A program that was supposed to help the National Security Agency pluck out electronic data crucial to the nation's safety is not up and running more than six years and $1.2 billion after it was launched, according to current and former government officials. The classified project, code-named Trailblazer, was promoted as the NSA's state-of-the-art tool for sifting through an ocean of modern-day digital communications and uncovering key nuggets to protect the nation against an ever-changing collection of enemies.

....Its main goal when it was launched in 1999 was to enable NSA analysts to connect the 2 million bits of data the agency ingests every hour ....The NSA initiative, which was designed to spot and analyze such hints, has resulted in little more than detailed schematic drawings filling almost an entire wall, according to intelligence experts familiar with the program. After an estimated $1.2 billion in development costs, only a few isolated analytical and technical tools have been produced, said an intelligence expert with extensive knowledge of the program. ...Trailblazer is "the biggest boondoggle going on now in the intelligence community," said Matthew Aid, who has advised three recent federal commissions and panels that investigated the Sept. 11 intelligence failures.

Consider the failure of the agency to pick up on more than 30 hits related to the September 11 attacks before they happened. And that this is the agency that Bush picked in 2001 to monitor Americans communicating with those overseas believed to be tied to Al Qaeda.

What does this tell you about Bush's warrantless NSA surveillance program, other than he violated our civil liberties and the FISA statute for nothing?

Although the Bush administration spent much of the past week defending the NSA's eavesdropping work as vital to keeping Americans safe from terrorism, virtually no attention has been paid to the agency's failure to deliver the system the NSA said was key to fulfilling that mission.

That means the government has been standing by while the agency has been gradually "going deaf" as unimportant communications drown out key pieces of information, a government official with extensive knowledge of Trailblazer told The Sun.

Who devised trailblazer in the first instance?

Trailblazer began as a signature program of Gen. Michael V. Hayden, who was the NSA's director from March 1999 until last spring. Early on, former officials familiar with the program said, it became clear to Hayden that the agency, with its rich history of developing cutting-edge technology, was falling behind the technology curve. He cast Trailblazer as the agency's future.

The same Michael Hayden who has been at the forefront of Bush's team trying to justify the warrantless surveillance. The same Michael Hayden who said the Fourth Amendment doesn't specify "probable cause" only "reasonable suspicion." From my earlier post based on the New York Times article on Hayden's speech defending the program:

.... "General Hayden defended the program's constitutionality. He said the lower, "reasonable belief" standard conformed to the wording of the Fourth Amendment, pointing out that it does not mention probable cause, but instead forbids "unreasonable" searches and seizures."

The last time I read the Fourth Amendment it said:

"The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."

This is a long article, but it's important to read the whole thing.

< Ezra's Hot, Kaine's Not | "We Don't Negoiate with Terrorists"; But We're Going To >
  • The Online Magazine with Liberal coverage of crime-related political and injustice news

  • Contribute To TalkLeft


  • Display: Sort:
    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#1)
    by ras on Sun Jan 29, 2006 at 11:59:09 PM EST
    TL, It tells me quite a lot, actually. The volumes are extremely low: 2m bits/hr = approx 200k bytes = 4.8M per hr = 115m per day = 40G per year. Whoa, that's low! I often download more than that myself, and many of you reading this do even more. Like, a single $100 Western Digital Drive could store the next years worth of data? Colossus lives! [Any geeks out there, have I screwed up my late-nite math on this?] All of which means they are simply NOT recording your conversations - well, not based on these numbers; there could be other factors not reported here - but the story looks like simple pattern-matching of selected items. It also looks grossly over budget. BTW, why has there not been a major terrorist attack on American soil since 9-11? The terrorists are well-manned, well-funded, and have lots of political support. Moreover they have managed to hit other targets (e.g. Spain) no probs. So why not in the US? Either the NSA (& related) efforts such as this are in fact enormously successful, and/or Bush's ME efforts are, or something else. Which do you choose? And why?

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#2)
    by Primus on Mon Jan 30, 2006 at 12:45:50 AM EST
    [Any geeks out there, have I screwed up my late-nite math on this?]
    Slightly, but not enough for it to make any difference. bits/8 = bytes, so the second number is 250kb. As one of those geeks, that 2 million bits number is unbelievably low. At the last ISP I worked for (a small regional ISP with a few thousand customers), I could have easily generated gigabytes of packet trace data in a day. At some of the larger ISPs and firms I've worked for in the past, it could be gigabytes/hour. I have to believe that the 2 million bits number is either a mistake or a misinterpretation. For the gov't to be unable to process that small amount of data in real-time means they're beyond incompetant, in that they have no idea what the heck they're doing and have never seen any of the number of open source packet analysis products out there that can easily handle that data load.

    I lost confidence in the article after
    All digital communications trapped by the NSA are transmitted to the agency's offices in computer codes of zeroes and ones.
    near the top of page two. I only skimmed after that, but the article doesn't seem to say anything more useful than an NSA project failed. I kept thinking "disinformation", but I've been messing about with my medications recently. Just for the pleasure of arguing:
    Either the NSA (& related) efforts such as this are in fact enormously successful, and/or Bush's ME efforts are, or something else. Which do you choose? And why?
    Something else, because of the length of time between the first and second attack on the Twin Towers. Al Qaeda is patient. I also expect they make contingency plans.

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#4)
    by ras on Mon Jan 30, 2006 at 01:51:11 AM EST
    Primus, The only way the govt could not process that amt of data - yeah, it's awfully small - is either fraud or incompetence or ... some sort of non-linear scalability problem, such as by xref'ing new data vs accumulated old data using a terrible - boy, do I mean terrible! - algorithm. Or some comb'n of these mistakes. I guess there's one other possibility, too - that the pgm is working great and the story was leaked as disinfo. But I tend to discount that cuz, unless some reporter was getting too close to uncovering a major success (horrors!), why leak at all? And why use such obviously low numbers?

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#5)
    by ras on Mon Jan 30, 2006 at 01:57:34 AM EST
    Allen, AQ is/was patient, but there's a time to hurry things along, too. They've been driven outta both Afg & Iraq, Libyta is no longer their friend. democracy is taking root in the ME ... etc. If they were gonna act, they shouldda done so already. They are prob still trying, and there may indeed be an attack in the future. But, even if so, the delay in the meantime looks to be more of necessity than choice. Me, I tend to lean to the theory that US troops on the borders of ME terror-sponsoring states are having an inhibiting effect. But the other theory, one I think has merit too, relates to the typical terrorist cell structure, wherein each member doesn't even know who's in his cell. Take out the central order-givers (AQ - "the base" in Arabic) as happened when AQ was clobbered in Afg, and the cells are then lost, waiting for orders than will never come. How can they regroup when they don't even know who their fellow members are?

    They've been driven outta both Afg & Iraq In your dreams, ras. If you're limited to four comments a day, why waste them posting stuff that everyone knows isn't true?

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#7)
    by Edger on Mon Jan 30, 2006 at 04:53:48 AM EST
    After an estimated $1.2 billion in development costs, only a few isolated analytical and technical tools have been produced [and] "the biggest boondoggle going on now in the intelligence community," [and] resulted in little more than detailed schematic drawings filling almost an entire wall
    This is a "Trailblazer" allright. Open ended budget. Open ended time frame. This is the wet dream of contract software developers. The programmers are creaming their jeans on this project, guaranteed. One of bushco's corporate buddies supplying the software engineering expertise? Standard SDLC (software development life cycle) is a seven stage process beginning with (1)"requirements capture" and ending with (6)"Solution Implementation" and (7)"Support and Maintenance". The drawings on the wall are the results of endless "meetings" which are part of stage (2)"Requirements Analysis" or maybe early stage (3)"Solution Design" at the most. And they've gotten this far in what, 8 years? This is paradise for the developers and analysts at somewhere in the neighborhood of probably 80-150k/year in salaries, or more. These guys and girls can hardly keep it in their pants. This project will never be completed. Completion is not in the interest of the people working on it. "biggest boondoggle"? That's the understatement of the year.

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#8)
    by Edger on Mon Jan 30, 2006 at 05:19:01 AM EST
    Of course, we still have Carnivore. And don't forget Echelon. And Tempest. and who knows what else. To keep the population, you know, "safe and secure". Remember the screens on the walls in Orwell's "1984"? You're sitting in front of one of them right now, reading this. Then there is Palladium and Altivore and all kinds of other "cool stuff".

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#9)
    by jimakaPPJ on Mon Jan 30, 2006 at 06:00:26 AM EST
    So now we shouldn't do it because it doesn't work..... ??? edger - You assume that is what the drawings are and take off from there. Please revisit the spelling of "assume." et al - If the numbers are at the front end of the funnel they look screwy. Now, if they are ar the output of the funnel.... 2mbs an hour processed and screened?

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#10)
    by Punchy on Mon Jan 30, 2006 at 06:38:26 AM EST
    BTW, why has there not been a major terrorist attack on American soil since 9-11? Because Bush has all the bad guys locked up in Gitmo. And there's some BAD ones in there...even the innocent (by the Admin's admission) ones.

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#11)
    by Edger on Mon Jan 30, 2006 at 06:51:42 AM EST
    Mike Ditto, if you're hovering on this one I'd love to hear your thoughts...

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#12)
    by Edger on Mon Jan 30, 2006 at 07:02:44 AM EST
    Jim: So now we shouldn't do it because it doesn't work..... ??? No, no, Jim. My point underneath my mildly facetious description of process, was the opposite. It will work. The developers will keep on producing "a few isolated analytical and technical tools". but they will do it as slowly as they can while keeping the customer happy with the progress. Drag it out as long as possible, make as much money as possible on it while dragging, all the while with the pot of gold in sight at the end of the rainbow: The "Support and Maintenance" stage. This is the most profitable, and also is the one stage that by design never ends. [sarcasm] They spend more and more of your money to be able to watch you ever closer and closer so they can get more and more of your money out of you to spend on watching you ever closer and closer, etc., etc., etc. :) Happy now? [/sarcasm]

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#13)
    by Punchy on Mon Jan 30, 2006 at 07:14:17 AM EST
    They spend more and more of your money to be able to watch you ever closer and closer so they can get more and more of your money out of you to spend on watching you ever closer and closer, etc., etc., etc. Ah, the oldest trick in the book. 2006: Why cut spending for a project you've already invested 1.2 BILL in, and it's almost done? 2007: Why cut spending for a system you've already invested....2 BILL? in, when it's almost ready and tested? 2008: cant quit now....look at all the potentially wasted money...gotta keep going...(which is also the Iraq strategy...sub "soliders" for "money")..... 2009: 3.5 Billion, but we're ALMOST THERE....

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#14)
    by Edger on Mon Jan 30, 2006 at 07:19:20 AM EST
    Punchy, And the poor confused donkey's hooves are wearing out and he's nearly dead on his feet following the carrot ("never fear, you'll be safe and secure!") dangling just out of reach in front of his nose...

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#15)
    by jimakaPPJ on Mon Jan 30, 2006 at 07:55:51 AM EST
    edger - Maybe, maybe not. It is true that engineers will "engineer" forever. That's why we have hardheaded Product Managers to control the budgets. ;-) BTW - Thoughts on 2mb "output" rather than input?

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#16)
    by Punchy on Mon Jan 30, 2006 at 08:35:17 AM EST
    Isn't this stunt the same one being pulled over at the TSA? That some state-of-the-art database/terrorist name screener/flight-pattern development software is years behind schedule, overbudget, and critically flawed? You're right about these software developers. When it comes to gov't, writing a great program for them writes you out of a contract. Writing a crappy one keeps you on the dole for years, at astronomical costs...

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#17)
    by Edger on Mon Jan 30, 2006 at 08:47:52 AM EST
    BTW - Thoughts on 2mb "output" rather than input? I think that the reporter made a mistake there, is not very technically knowledgable... a misquote of something s/he heard?
    "This system's purpose would be to monitor communications and detect would-be terrorists and plots before they happen... This project is not interested in funding "evolutionary" changes in technology, e.g., bit-step improvements to current data mining and storage techniques. Rather, the amount of data that the directors are anticipating (petabytes!) would require massive leaps in technology (and perhaps also some massive leaps in surveillance laws). According to DARPA, such data collection "increases information coverage by an order of magnitude," and ultimately "requires keeping track of individuals and understanding how they fit into models." --The new technology at the root of the NSA wiretap scandal


    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#18)
    by ras on Mon Jan 30, 2006 at 09:44:17 AM EST
    Edger, I've seen info projects run over like this before. Usually, it's incompetence on the part of the managers, and the contractors just keep smiling and billing and doing their jobs (well, at a higher level; the lower-level techs will be as frustrated as anyone). As for the 7-stage SDLC process, that presumes that scope & requirements are properly in place: i.e. that the govt actually knows what it wants. But this might not be the case: they might be prototyping, essentially meaning that this is really an open-ended research project disguised as a computer system. That'd also explain the low-volumes: it's alpha-test data, no more, and they're not even up to proper beta- and/or stress-testing yet. Then again, maybe they're just laundering funds to be spent elsewhere, eh?

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#19)
    by Edger on Mon Jan 30, 2006 at 12:28:18 PM EST
    Punchy:
    Isn't this stunt the same one being pulled over at the TSA?
    Yep... exactly. Because the whole point of these things is fearmongering Americans, not catching "terrists"
    When it comes to gov't, writing a great program for them writes you out of a contract.
    Happens all the time in software development. As in all other business it's easier and cheaper to sell to an existing customer with whom you have an ongoing relationship than to find a new customer. Less acquisition cost, less overhead, more profit.

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#20)
    by Edger on Mon Jan 30, 2006 at 04:15:20 PM EST
    Jim: edger - Maybe, maybe not. It is true that engineers will "engineer" forever. That's why we have hardheaded Product Managers to control the budgets. ;-) Well, you can write them and thank them. They've taken $1.2 Billion Dollars of tax money (so far, and they're just getting started) and poured it straight down the drain for you. Yep, that good ole' George is a hell of a guy, ain't he?
    Then there is the matter of encryption technologies that can turn even intercepted communications into gobbledygook. "The odds are nigh on impossible that the NSA or anybody else is going to be able to break" an encrypted message, says security expert and author Schwartau. Another technology that Osama bin Laden s minions reportedly used falls under the rubric of steganography: cloaking one type of data file within another. It is possible, for example, to hide a text file with attack plans within a bit-mapped photo of Britney Spears. Just try to filter down the number of those images flying around the Internet. And even the most advanced spying technology can be stymied by embarrassingly primitive countermeasures... "To use a crude example: maybe the terrorists substituted the word 'banana' for 'bomb' and 'orange' for 'World Trade Center.' Do you flag every unusual pattern with random associations?" Even if the obstacles of bureaucracy, societal resistance and technical limitations were all to be surmounted, there's no assurance that high-tech spyware would ever provide the kind of security that people now crave. Will these technologies help recognize the danger next time? Even the most sophisticated intelligence paraphernalia still can't guarantee success when pitted against the malevolent combination of human ingenuity and capacity for evil.


    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#21)
    by rMatey on Mon Jan 30, 2006 at 05:57:53 PM EST
    .."an ever-changing collection of enemies..." But mostly liberals, democrats and anyone else who doesn't believe that this moron isn't fit to be king.

    Re: NSA Electronic Data Surveillance: Doomed From (none / 0) (#22)
    by jimakaPPJ on Tue Jan 31, 2006 at 11:28:07 AM EST
    edger - My comment was non-political, so why did I know you would respond with politics? BTW - Do some reading on the Church Committee if you want to understand when the decline of the CIA started.