Saturday, June 08, 2019

The Phoenix Lights, 1945

From John Keeling, by way of the May 2019 Fortean Times (p. 28):
In 1945 a jittery American public was mistaking Venus for Japan’s FU-GO balloon bombs on an alarmingly regular basis. 9,000 of the 30 ft balloons with incendiary bomb payloads had been launched against the US in the hope of causing large-scale forest fires and spreading terror....On June 6th, Phoenix and several other Arizona communities had their first ‘Jap balloon’ panic. Telephone lines to the press, police department, sheriff’s office and weather bureau were reportedly jammed....Luke Field and Williams Field fliers, checking the object from planes, were able to report back definitely that there was no balloon where reported. And Phoenix Junior college’s 5 inch refractor telescope clearly identified the object as Venus. According to the Associated Press, Tucson had the same experience, with Davis-Monthan fliers being ‘sent to cut down the invader.’

Tuesday, January 01, 2019

Books read in 2018

Not much blogging going on here still, but here's my annual list of books read for 2018.
  • Charles Arthur, Cyber Wars: Hacks that Shocked the Business World
  • Radley Balko and Tucker Carrington, The Cadaver King and the Country Dentist: A True Story of Injustice in the American South
  • Mary Beard, SPQR: A History of Ancient Rome
  • Yochai Benkler, Robert Faris, and Hal Roberts, Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics
  • Ronen Bergman, Rise and Kill First: The Secret History of Israel's Targeted Assassinations
  • Rebecca Burns and David Dayen, Fat Cat: The Steve Mnuchin Story
  • John Carreyrou, Bad Blood: Secrets and Lies in a Silicon Valley Startup
  • Graydon Carter, George Kalogerakis, and Kurt Andersen, Spy: The Funny Years
  • Stephen Ellis, This Present Darkness: A History of Nigerian Organized Crime
  • Jason Fagone, The Woman Who Smashed Codes: A True Story of Love, Spies, and the Unlikely Heroine Who Outwitted America's Enemies
  • Paul French, City of Devils: The Two Men Who Ruled the Underworld of Old Shanghai
  • Diego Gambetta, Codes of the Underworld: How Criminals Communicate
  • Robert M. Gates, Duty: Memoirs of a Secretary at War
  • Atul Gawande, Being Mortal: Medicine and What Matters in the End
  • David Golumbia, The Politics of Bitcoin: Software as Right-Wing Extremism
  • Richards J. Heuer Jr. and Randolph H. Pherson, Structured Analytic Techniques for Intelligence Analysis
  • Michael Isikoff and David Corn, Russian Roulette: The Inside Story of Putin's War on America and the Election of Donald Trump
  • Sarah Jeong, The Internet of Garbage
  • Steven Johnson, Farsighted: How We Make the Decisions That Matter the Most
  • Louise M. Kaiser and Randolph H. Pherson, Analytic Writing Guide
  • Chuck Klosterman, But What If We're Wrong?: Thinking About the Present As If It Were the Past
  • Susan Landau, Listening In: Cybersecurity in an Insecure Age
  • Peter T. Leeson, WTF?! An Economic Tour of the Weird
  • Jeffrey Lewis, The 2020 Commission Report on the North Korean Nuclear Attacks Against the United States
  • Michael Lewis, The Fifth Risk
  • Liliana Mason, Uncivil Agreement: How Politics Became Our Identity
  • Nick Mason, Inside Out: A Personal History of Pink Floyd (new updated 2017 edition)
  • Tim Maurer, Cyber Mercenaries: The State, Hackers, and Power
  • Jefferson Morley, The Ghost: The Secret Life of CIA Spymaster James Jesus Angleton
  • Roger Naylor, The Amazing Kolb Brothers of Grand Canyon
  • Helen Nissenbaum, Privacy in Context: Technology, Policy, and the Integrity of Social Life
  • Ellen Pao, Reset: My Fight for Inclusion and Lasting Change
  • Dana Richards, editor, Dear Martin/Dear Marcello: Gardner and Truzzi on Skepticism
  • Louis Rossetto, Change Is Good: A Story of the Heroic Era of the Internet (1st edition, #1453, Kickstarter)
  • David E. Sanger, The Perfect Weapon: War, Sabotage, and Fear in the Cyber Age
  • Eli Saslow, Rising Out of Hatred: The Awakening of a Former White Nationalist
  • Harold Schechter, The Pirate (Amazon Prime Reading "Bloodlands Collection")
  • Harold Schechter, Little Slaughterhouse on the Prairie (Amazon Prime Reading "Bloodlands Collection")
  • Harold Schechter, The Brick Slayer (Amazon Prime Reading "Bloodlands Collection")
  • Harold Schechter, Panic (Amazon Prime Reading "Bloodlands Collection")
  • Harold Schechter, Rampage (Amazon Prime Reading "Bloodlands Collection")
  • Harold Schechter, The Pied Piper (Amazon Prime Reading "Bloodlands Collection")
  • Natasha Dow Schüll, Addiction by Design: Machine Gambling in Las Vegas
  • Kevin Simler and Robin Hanson, The Elephant in the Brain: Hidden Motives in Everyday Life
  • P.W. Singer and Emerson T. Brooking, LikeWar: The Weaponization of Social Media
  • Ali Soufan, Anatomy of Terror: From the Death of Bin Laden to the Rise of the Islamic State
  • Robert Timberg, The Nightingale's Song (bio of John McCain, James Webb, Oliver North, Robert McFarlane, and John Poindexter)
  • Mick West, Escaping the Rabbit Hole: How to Debunk Conspiracy Theories Using Facts, Logic, and Respect
  • Rick Wilson, Everything Trump Touches Dies: A Republican Strategist Gets Real About the Worst President Ever
  • Michael Wolff, Fire and Fury: Inside the Trump White House
  • Bob Woodward, Fear: Trump in the White House
  • Tim Wu, The Curse of Bigness: Antitrust in the New Gilded Age
I made some progress on a few other books:
  • Herbert Asbury, The Barbary Coast: An Informal History of the San Francisco Underworld (will probably finish today)
  • Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt
  • Douglas Hofstadter and Emmanuel Sander, Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
Top for 2018:  Singer and Brooking, Bergman, Balko and Carrington, Gawande, Carreyrou, Sanger, Simler and Hanson, Soufan, Isikoff and Corn, Fagone, French, Schüll, Michael Lewis, Mason, Benkler et al., West, Wu, Saslow, Naylor. I didn't care for the Klosterman book at all--quick read, but a waste of time.

(Previously: 2017201620152014201320122011201020092008200720062005.)

Monday, January 01, 2018

Books read in 2017

Not much blogging going on here still, but here's my annual list of books read for 2017. Items with hyperlinks are linked directly to the item online (usually PDF, some of these are reports rather than books, though I've made no attempt to collect all papers, blog posts, and reports I read here), with no paywall or fee.
  • Lilian Ablon, Andy Bogart, Zero Days, Thousands of Nights: The Life and Times of Zero-Day Vulnerabilities and Their Exploits
  • Ben Buchanan, The Cybersecurity Dilemma: Hacking, Trust and Fear Between Nations
  • J.D. Chandler, Hidden History of Portland, Oregon
  • Ted Conover, Newjack: Guarding Sing Sing
  • Richard A. Clarke and R.P. Eddy, Warnings: Finding Cassandras to Stop Catastrophes
  • Thomas H. Davenport and Julia Kirby, Only Humans Need Apply: Winners & Losers in the Age of Smart Machines
  • Mike Edison, Dirty, Dirty, Dirty: Of Playboys, Pigs, and Penthouse Paupers--An American Tale of Sex and Wonder
  • FINRA, Distributed Ledger Technology: Implications of Blockchain for the Securities Industry
  • Al Franken, Al Franken, Giant of the Senate
  • David Gerard, Attack of the 50 Foot Blockchain: Bitcoin, Blockchain, Ethereum & Smart Contracts
  • Joscelyn Godwin, Upstate Cauldron: Eccentric Spiritual Movements in Early New York State
  • Jonathan Goldsmith, Stay Interesting: I Don't Always Tell Stories About My Life, But When I Do They're True and Amazing
  • Heidi Grant Halvorson, No One Understands You: And What To Do About It
  • Jon Lindsay, Tai Ming Cheung, and Derek S. Reveron, editors, China and Cybersecurity: Espionage, Strategy, and Politics in the Digital Domain
  • William MacAskill, Doing Good Better: Effective Altruism and How You Can Make a Difference
  • Jane Mayer, Dark Money: The Hidden History of the Billionaires Behind the Rise of the Radical Right
  • Nick Middleton, An Atlas of Countries That Don't Exist: A Compendium of Fifty Unrecognized and Largely Unnoticed States
  • Kevin Mitnick, The Art of Invisibility: The World's Most Famous Hacker Teaches You How to Be Safe in the Age of Big Brother and Big Data
  • Andrew Monaghan, "The New Russian Foreign Policy Concept: Evolving Continuity," Chatham House, 2013 (PDF)
  • Milton Mueller, Will the Internet Fragment? Sovereignty, Globalization and Cyberspace
  • Tom Nichols, The Death of Expertise: The Campaign Against Established Knowledge and Why it Matters
  • David Ronfeldt, Beware the Hubris-Nemesis Complex: A Concept for Leadership Analysis
  • Thomas Rid, Rise of the Machines: A Cybernetic History
  • Gabriel Sherman, The Loudest Voice in the Room: How the Brilliant, Bombastic Roger Ailes Built Fox News--and Divided a Country
  • Doug Stanhope, Digging Up Mother: A Love Story
  • Doug Stanhope, This Is Not Fame: A "From What I Re-Memoir"
  • Charles Stross, Halting State
  • Charles Stross, Rule 34
  • Sarah Vowell, Unfamiliar Fishes
  • Timothy Walton, Challenges in Intelligence Analysis: Lessons from 1300 BCE to the Present
  • Kristan J. Wheaton and Melonie K. Richey, Strawman
  • Ilya Zaslavskiy, How Non-State Actors Export Kleptocratic Norms to the West (PDF)
I may or may not have made progress on a few other books (first four from 2017, next two from 2016, one from 2015,  next three from 2014, next three from 2013, last two still not finished from 2012--I have trouble with e-books, especially very long nonfiction e-books):
  • Helen Nissenbaum, Privacy in Context: Technology, Policy, and the Integrity of Social Life
  • Dana Richards, editor, Dear Martin/Dear Marcello: Gardner and Truzzi on Skepticism
  • Richards J. Heuer, Jr., Structured Analytics Techniques for Intelligence Analysis
  • Louis M. Kaiser, Analytic Writing Guide
  • Andreas Antonopoulos, Mastering Bitcoin: Unlocking Digital Cryptocurrencies (now 2nd ed)
  • Robert M. Gates, Duty: Memoirs of a Secretary at War
  • John Searle, Making the Social World
  • Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt
  • Massimo Pigliucci and Maarten Boudry, Philosophy of Pseudoscience: Reconsidering the Demarcation Problem
  • Steven Pinker, The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century
  • Richard Bejtlich, The Practice of Network Security Monitoring
  • James Grimmelmann, Internet Law: Cases & Problems (v2; v3 is out now)
  • Douglas Hofstadter and Emmanuel Sander, Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
  • Mark Dowd, John McDonald, and Justin Schuh, The Art of Software Security Assessment: Identifying and Avoiding Software Vulnerabilities
  • Michal Zalewski, The Tangled Web: A Guide to Securing Modern Web Applications
Top for 2017:  Rid, Buchanan, Sherman, Mayer, Clarke and Eddy, Conover, Middleton.

I completed three Coursera courses in 2017, two of which I recommend:


(Previously: 201620152014201320122011201020092008200720062005.)

Sunday, March 12, 2017

Rep. Tom Graves' Active Cyber Defense Certainty Act

Rep. Tom Graves (R-GA14) has circulated a draft bill, the "Active Cyber Defense Certainty Act" (or ACDC Act), which amends the Computer Fraud and Abuse Act (18 USC 1030) to legalize certain forms of "hacking back" for the purposes of collecting information about an attacker in order to facilitate criminal prosecution or other countermeasures.

The bill as it currently stands is not a good bill, for the following reasons:

1. It ignores the recommendations in a recent report, "Into the Gray Zone: Active Defense by the Private Sector Against Cyber Threats," from the Center for Cyber & Homeland Security at the George Washington University. This report distinguishes between low-risk active defense activities within the boundaries of the defender's own network, such as the use of deceptive technology (honeypots, honeynets, tarpitting), the use of beaconing technology to provide notifications in case of intrusions, and research in deep and dark web underground sites, on the one hand, and higher-risk active defense activities such as botnet takedowns, sanctions and indictments, white-hat ransomware, and rescue missions to recover stolen assets, on the other. One of the report's key questions for an active defense measure is "is the active defense measure authorized, whether by an oversight body, law enforcement, or the owner of the affected network?"  This bill creates no mechanism for providing particular authorizations (also see points 2 and 3, below).

The "Into the Gray Zone" report also suggests that if a decision is made to authorize the accessing of a remote system (an attacker's system is almost always the system of another victim) for information collection purposes, it should be limited to cases in which a defender can "assert a positive identification of the hostile actor with near certainty, relying on multiple credible attribution methods." This, however, seems too strict a condition to impose.

Finally, however, this report advises that, even without a change in the law, DOJ "should exercise greater discretion in choosing when to enforce the CFAA and other relevant laws, and should provide clarity about how it intends to exercise such discretion. Companies engaging in activities that may push the limits of the law, but are intended to defend corporate data or end a malicious attack against a private server should not be prioritized for investigation or prosecution." (p. 28) The report cites active defense activity by Google in response to hacking from China as an example where there was no prosecution or sanction for accessing remote systems being used by attackers. This proposal seems to me a wiser course of action than adopting this bill. (Also see point 5, below.)

2. It disregards the recommendations from the Center for Strategic and International Studies Cyber Policy Task Force on the subject of active defense. The CSIS Cyber Policy Task Force report contains a short three-paragraph section on active defense (p. 14) which throws cold water on the idea, calling active defense "at best a stopgap measure, intended to address companies’ frustration over the seeming impunity of transborder criminals" and affirming that only governments should be authorized to engage in activities on the high-risk side, and that it is their responsibility to coordinate and engage in such activity. It does offer up a possibility for a proposal that allows accessing remote systems by private parties in its last sentence: "Additionally, the administration could consider measures, carried out with the prior approval of federal law enforcement agencies (most likely requiring a warrant to enter a third-party network) to recover or delete stolen data stored on servers or networks under U.S. jurisdiction." This bill does not require approval from federal law enforcement agencies or a warrant for accessing remote systems or networks, and jurisdiction is only implicit.

3. While the proposal in the bill resembles a proposal made in a Mercatus Center at George Mason University proposal by Anthony Glosson, it adopts the carrot element of the proposal while neglecting the stick. Glosson's proposal is that, like this bill, private parties should be permitted to access remote attacking systems in order to collect information ("observation and access"), but not to engage in "disruption and destruction." However, Glosson suggests three requirements be present to make such access and information collection permissible, and if those requirements are not present, that there be "stiff statutory damages" imposed. The bill omits any statutory damages, and imposes only one of Glosson's three requirements (though a previous version of the bill included the second). Glosson's three requirements are (1) that the defender's actions are limited to observation and access, (2) that the attacker was routing traffic through the defender's network at the time of the active defense action, and (3) that obtaining the owner of the attacking system's cooperation at the time of the attack was impractical.  This third criterion is a critical one, and a good way to observe the undesirability of this bill is to imagine that you are the owner of the intermediary system used by the attacker to go after a third party--what would you want that third party to be able to do with your system without your permission or consent?

4. The bill appears to have been somewhat hastily written and sloppily updated, failing to update a persistent typographical error ("the victim' [sic] own network") through its revisions, and the current version seems to be somewhat incoherent. In its current form it is unlikely to meet its short title objective of encouraging certainty.

The current version of the bill makes it legal for a victim of a "persistent unauthorized intrusion" to access "without authorization the computer of the attacker to the victim' [sic] own network to gather information in order to establish attribution of criminal activity to share with law enforcement or to disrupt continued unauthorized activity against the victim's own network," so long as this does not destroy information on the system, cause physical injury, or create a threat to public health or safety.

The phrase "without authorization the computer of the attacker to the victim's own network" doesn't make sense [it should say "attacker of" or "attacker against"], and appears to be the result of poor editing from the prior version of the bill, which made permissible accessing "without authorization a computer connected to the victim' [sic] own network", with the rest of the text remaining the same. This prior wording apparently attempted to thread the needle of the GWU "Into the Gray Zone" report by defining the accessing of a remote system as being within the boundaries of the defender's own network, and thus on the low-risk side of the equation. However, the wording "connected to the victim's own network" is ambiguous and unclear--does it mean directly connected (e.g., to a WiFi access point or LAN port on a switch), in which case this is much less useful, or does it mean any active session flow of packets over the Internet into the victim's network (similar to Glosson's second requirement)? The latter is the more reasonable and charitable interpretation, but it should be made more explicit and could perhaps be too strict--what happens if the attacker disconnects just moments before the active defense activity begins?

Left unsaid in the bill is what can be done with information collected from the attacking system, which might include information belonging to other victims, the exposure of which could cause harm. Presumably other remedies from other statutes would exist if a defender engaged in such exposure, but it seems to me that this bill would be improved by making the parameters of permissible action more explicit and restrictive. Perhaps the current wording limits actions to information sharing with law enforcement and reconfiguration of one's own defensive systems based on the collected information, but "to disrupt continued unauthorized activity against the victim's own network" is a purpose that could be achieved by a much broader set of actions, which could cause harm to other victims.

5. It's not clear that the bill is necessary, given that security researchers are today (as they have been for years) taking steps to access infrastructure used by malicious cyber threat actors in order to monitor their activity and collect intelligence information. They are already making legal and regulatory risk decisions which incorporate the existing CFAA, and deciding to proceed anyway.

If this bill is to move forward, it needs some additional work.

(News story on the bill: Michael Mimoso, "Active Defense Bill Raises Concerns of Potential Consequences," ThreatPost.
Further reading: Paul Rosenzweig, "A Typology for Evaluating Active Cyber Defenses," Lawfare blog)

UPDATE (March 14, 2017): Robert Chesney wrote a good critique of the bill at the Lawfare blog, "Legislative Hackback: Notes on the Active Cyber Defense Certainty Act discussion draft," in which he points out that the word "persistent" is undefined and vague, notes that "intrusion" excludes distributed denial of service attacks from permissible cases of response under this bill, and wisely notes that there may be multiple computers in an attack chain used by the attacker, while the bill is written as though there is only one.  (It is also noteworthy that an attacking IP could be a firewall in front of an attacking machine, and a response attempting to connect to that IP could be redirected to a completely different system.)  Chesney also questions whether destroying information is the right limit on responsive activity, as opposed to altering information (such as system configurations). He also notes that the restrictions for destruction, physical injury, and threats to public health and safety are probably insufficient, noting as I did above that there could be other forms of harm from disseminating confidential information discovered on the attacking system.

I think a more interesting bill that would create incentives for companies to invest in security and to appropriately share information about attacks (rather than trying to hide it) would be a bill that created a safe harbor or liability limits for a company whose systems are used to attack third parties, if they have taken certain precautionary measures (such as having patched all known vulnerabilities more than 30 days old, and having a continuous monitoring program) and if they also share in a timely manner information about their breach.

UPDATE (May 25, 2017): Rep. Graves has released a version 2.0 of his bill which is vastly improved, addressing almost all of my concerns above. The new Sec. 2 of the bill puts the use beaconing technology on a sound legal footing, which is consistent with the recommendations of the CSIS "Into the Gray Zone" report. The new Sec. 4 of the bill requires notification of the FBI, which, while it isn't the notification of/deferral to organizations which have their own cyber defense teams to protect and investigate their own compromised infrastructure, it might effectively serve the same purpose, and it also provides a deterrent to irresponsible active defense.  The core of the former bill, Sec. 3, has been revised to limit what can be done, so that now taking or exposing content on the attacker machine belonging to other parties would not be permissible. And there is also a new Sec. 5 of the bill, which sunsets it after two years. I cautiously support the new bill as a potentially useful experiment.

UPDATE (October 14, 2017): A new version of the bill was released this week which has further improvements. Instead of just creating an exemption to the CFAA, it creates a defense to a criminal charge, and makes clear that it is not a defense for civil liability. This means if you are within the bounds of the new rules accessing the systems of a third party which is another victim of the attacker, you won't go to jail for it, but you could still be successfully sued for damages by that third party. The new version of the bill also lists a few more things which you are NOT permitted to do in order to use the defense, and it requires that the FBI create a program for receiving advance notices from individuals and organizations that intend to use these measures, as well as a requirement for an annual assessment of this legislation's effectiveness.

UPDATE (February 2, 2018): There are still a few issues with the current version of the Graves bill. (1) It doesn't require defenders to document and disclose actions taken against systems not owned by the attacker to the owners of those systems. (2) It places no limits on what vulnerabilities may be exploited on intermediary or attacker systems. (3) It allows destructive actions against information which belongs to the defender, as well as against any information or system which belongs to the attacker. (4) It does not limit the targets to systems within U.S. jurisdiction, or does it require any judicial approval. Attacks on systems outside U.S. jurisdiction could result in state-sponsored blowback. (5) The exception to permitted activity for any action which "intentionally results in intrusive or remote access into an intermediary's computer" seems at odds with the overall proposal, since 90%+ of the time the systems used by attackers will belong to an intermediary. (6) Sec. 5's requirement that the FBI be notified and presented with various pieces of information prior to the active defense seems both too strict and too loose. Too strict in that it doesn't allow pre-certification and must occur in the course of an attack, too loose in that it requires that the FBI acknowledge receipt before proceeding but no actual approval or certification, and that there's a loophole on one of the required pieces of information to be given to the FBI, which is any other information requested by the FBI for the purposes of oversight. Since all the active defender requires is acknowledgment of receipt, if the FBI doesn't request any such further information as part of that acknowledgement, the defender is good to go immediately at that point before any further information is provided. Sec. 5 is kind of a fake certification process--there is no actual certification or validation process that must occur.

Thursday, February 16, 2017

Confusing the two Trump cybersecurity executive orders

In Andy Greenberg's Wired article on February 9, 2017, "Trump Cybersecurity Chief Could Be a 'Voice of Reason," he writes:
But when Trump’s draft executive order on cybersecurity emerged last week, it surprised the cybersecurity world by hewing closely to the recommendations of bipartisan experts—including one commission assembled by the Obama administration.
The described timing and the link both refer to the original draft cybersecurity executive order, which does not at all resemble the recommendations of Obama's Commission on Enhancing National Cybersecurity or the recommendations of the Center for Strategic and International Studies Cyber Policy Task Force, which both included input from large numbers of security experts. Contrary to what Greenberg says, the executive order he refers to was widely criticized on a number of grounds, including that it is incredibly vague and high level, specifies an extremely short time frame for its reviews, and that it seemed to think it was a good idea to collect information about major U.S. vulnerabilities and defenses into one place and put it into the hands of then-National Security Advisor Michael T. Flynn. That original version of the executive order resembled the Trump campaign's website policy proposal on cybersecurity.

The positive remarks, instead, were for a revised version of the cybersecurity executive order which was verbally described to reporters on the morning of January 31, the day that the signing of the order was expected to happen at 3 p.m., after Trump met for a listening session with security experts. The signing was cancelled, and the order has not yet been issued, but a draft subsequently got some circulation later in the week and was made public at the Lawfare blog on February 9.

This executive order contains recommendations consistent with both the Cybersecurity Commission report and the CSIS Cyber Policy Task Force report, mandating the use of the NIST Cybersecurity Framework by federal agencies, putting the Office of Management and Budget (OMB) in charge of enterprise risk assessment across agencies, promoting IT modernization and the promotion of cloud and shared services infrastructure, and directing DHS and other agency heads to work with private sector critical infrastructure owners on defenses.

One key thing it does not do, which was recommended by both reports, is elevate the White House cybersecurity coordinator role (a role which the Trump administration has not yet filled, which was held by Michael Daniel in the Obama administration) to an Assistant to the President, reflecting the importance of cybersecurity. Greenberg's piece seems to assume that Thomas Bossert is in the lead cybersecurity coordinator role, but his role is Homeland Security Advisor (the role previously held by Lisa Monaco in the Obama administration), with broad responsibility for homeland security and counterterrorism, not cybersecurity-specific.

Despite Greenberg's error confusing the two executive orders being pointed out to him on Twitter on February 9, the article hasn't been corrected as of February 16.

Sunday, January 01, 2017

Books read in 2016

Not much blogging going on here still, but here's my annual list of books read for 2016. Items with hyperlinks are linked directly to the item online (usually PDF, some of these are reports rather than books), with no paywall or fee.
  • Andreas Antonopoulos, The Internet of Money
  • Herbert Asbury, The Gangs of New York: An Informal History of the Underworld
  • Rob Brotherton, Suspicious Minds: Why We Believe Conspiracy Theories
  • Center for Cyber & Homeland Security, Into the Gray Zone: The Private Sector and Active Defense Against Cyber Threats
  • Michael D'Antonio, Never Enough: Donald Trump and the Pursuit of Success
  • Henning Diedrich, Ethereum: Blockchains, Digital Assets, Smart Contracts, Decentralized Autonomous Organizations
  • Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future
  • Emma A. Jane and Chris Fleming, Modern Conspiracy: The Importance of Being Paranoid
  • Roger Z. George and James B. Bruce, editors, Analyzing Intelligence: Origins, Obstacles, and Innovations
  • Peter Gutmann, Engineering Security
  • House Homeland Security Committee, Going Dark, Going Forward: A Primer on the Encryption Debate
  • Dr. Rob Johnston, Analytic Culture in the U.S. Intelligence Community: An Ethnographic Study
  • R.V. Jones, Most Secret War
  • Fred Kaplan, Dark Territory: The Secret History of Cyber War
  • Maria Konnikova, The Confidence Game: Why We Fall for It...Every Time
  • Adam Lee, hilarious blog commentary on Atlas Shrugged
  • Deborah Lipstadt, Denying the Holocaust: The Growing Assault on Truth and Memory
  • Dan Lyons, Disrupted: My Misadventure in the Startup Bubble
  • Geoff Manaugh, A Burglar's Guide to the City
  • Felix Martin, Money: The Unauthorized Biography--From Coinage to Cryptocurrencies
  • Nathaniel Popper, Digital Gold: Bitcoin and the Inside Story of the Misfits and Millionaires Trying to Reinvent Money
  • John Allen Paulos, A Numerate Life: A Mathematician Explores the Vagaries of Life, His Own and Probably Yours
  • Mary Roach, Grunt: The Curious Science of Humans at War
  • Jon Ronson, The Elephant in the Room: A Journey into the Trump Campaign and the "Alt-Right"
  • Oliver Sacks, On the Move: A Life
  • Luc Sante, Low Life: Lures and Snares of Old New York
  • Adam Segal, The Hacked World Order: How Nations Fight, Trade, Maneuver, and Manipulate in the Digital Age
  • Steve Silberman, NeuroTribes: The Legacy of Autism and the Future of Neurodiversity
  • Richard Stiennon, There Will Be Cyberwar: How the Move to Network-Centric War Fighting Has Set the Stage for Cyberwar
  • Russell G. Swenson, editor, Bringing Intelligence About: Practitioners Reflect on Best Practices
  • U.S. Army Special Operations Command, "Little Green Men": A Primer on Modern Russian Unconventional Warfare, Ukraine, 2013-2014
  • Joseph E. Uscinski and Joseph M. Parent, American Conspiracy Theories
  • Paul Vigna and Michael J. Casey, The Age of Crypto Currency: How Bitcoin and the Blockchain Are Challenging the Global Economic Order
I made progress on a few other books (first four from 2016, one from 2015,  next three from 2014, next three from 2013, last two still not finished from 2012--I have trouble with e-books, especially very long nonfiction e-books):
  • Andreas Antonopoulos, Mastering Bitcoin: Unlocking Digital Cryptocurrencies
  • Robert M. Gates, Duty: Memoirs of a Secretary at War
  • Jocelyn Godwin, Upstate Cauldron: Eccentric Spiritual Movements in Early New York State
  • Thomas Rid, Rise of the Machines: A Cybernetic History
  • John Searle, Making the Social World
  • Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt
  • Massimo Pigliucci and Maarten Boudry, Philosophy of Pseudoscience: Reconsidering the Demarcation Problem
  • Steven Pinker, The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century
  • Richard Bejtlich, The Practice of Network Security Monitoring
  • James Grimmelmann, Internet Law: Cases & Problems (v2; v3 is out now)
  • Douglas Hofstadter and Emmanuel Sander, Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
  • Mark Dowd, John McDonald, and Justin Schuh, The Art of Software Security Assessment: Identifying and Avoiding Software Vulnerabilities
  • Michal Zalewski, The Tangled Web: A Guide to Securing Modern Web Applications
Top ten for 2016:  Sacks, Silberman, Jane & Fleming, Konnikova, Manaugh, Lyons, Popper, Uscinski & Parent, Jones, Lipstadt.

(Previously: 20152014201320122011201020092008200720062005.)

Friday, January 01, 2016

Books read in 2015

Not much blogging going on here lately, but here's my annual list of books read for 2015:
  • George A. Akerlof and Robert J. Shiller, Phishing for Phools: The Economics of Manipulation & Deception
  • Jeffrey S Bardin, The Illusion of Due Diligence: Notes from the CISO Underground
  • Bill Browder, Red Notice: A True Story of High Finance, Murder, and One Man's Fight for Justice
  • Ron Chernow, Alexander Hamilton
  • Gabriella Coleman, Hacker, Hoaxer, Whistleblower, Spy: The Many Faces of Anonymous
  • Karen Dawisha, Putin's Kleptocracy: Who Owns Russia?
  • Laura DeNardis, The Global War for Internet Governance
  • Daniel C. Dennett and Linda LaScola, Caught in the Pulpit: Leaving Belief Behind
  • Mary Douglas and Aaron Wildavsky, Risk and Culture: An Essay on the Selection of Technological and Environmental Dangers
  • William J. Drake and Monroe Price, editors, Internet Governance: The NETmundial Roadmap
  • Jon Friedman and Mark Bouchard, Definitive Guide to Cyber Threat Intelligence
  • Marc Goodman, Future Crimes: Everything is Connected, Everyone is Vulnerable, and What We Can Do About It
  • Marc Hallet, A Critical Appraisal of George Adamski: The Man Who Spoke to the Space Brothers
  • Shane Harris, @War: The Rise of the Military-Internet Complex
  • Peter T. Leeson, The Invisible Hook: The Hidden Economics of Pirates
  • Reed Massengill, Becoming American Express: 150 Years of Reinvention and Customer Service
  • James Andrew Miller and Tom Shales, Live From New York: The Complete, Uncensored History of Saturday Night Live, as Told By Its Stars, Writers, and Guests (two new chapters)
  • David T. Moore, Critical Thinking and Intelligence Analysis
  • Richard E. Nisbett, Mindware: Tools for Smart Thinking
  • Tony Ortega, The Unbreakable Miss Lovely: How the Church of Scientology Tried to Destroy Paulette Cooper
  • Whitney Phillips, This is Why We Can't Have Nice Things: Mapping the Relationship Between Online Trolling and Mainstream Culture
  • Joseph M. Reagle, Jr., Reading the Comments: Likers, Haters, and Manipulators at the Bottom of the Web
  • Jon Ronson, Lost at Sea: The Jon Ronson Mysteries
  • Jon Ronson, So You've Been Publicly Shamed
  • Bruce Schneier, Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World
  • P.W. Singer and Allan Friedman, Cybersecurity and Cyberwar: What Everyone Needs to Know
  • David Skarbek, The Social Order of the Underworld: How Prison Gangs Govern the American Penal System
  • Andrei Soldatov and Irina Borogan, The Red Web: The Struggle Between Russia's Digital Dictators and the New Online Revolutionaries
  • Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction
  • Richard H. Thaler, Misbehaving: The Making of Behavioral Economics
I made progress on a few other books (first two last year,  next four from 2014, next three from 2013, last two still not finished from 2012--I have trouble with very long nonfiction e-books):
  • Roger Z. George and James B. Bruce, editors, Analyzing Intelligence: Origins, Obstacles, and Innovations
  • John Searle, Making the Social World
  • Peter Gutmann, Engineering Security
  • Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt
  • Massimo Pigliucci and Maarten Boudry, Philosophy of Pseudoscience: Reconsidering the Demarcation Problem
  • Steven Pinker, The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century
  • Richard Bejtlich, The Practice of Network Security Monitoring
  • James Grimmelmann, Internet Law: Cases & Problems (v2; v3 is out now)
  • Douglas Hofstadter and Emmanuel Sander, Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
  • Mark Dowd, John McDonald, and Justin Schuh, The Art of Software Security Assessment: Identifying and Avoiding Software Vulnerabilities
  • Michal Zalewski, The Tangled Web: A Guide to Securing Modern Web Applications
Top ten for 2015:  Browder, Chernow, Coleman, Ronson (Shamed), Schneier, Phillips, Nisbett, Ortega, Miller and Shales, Thaler. I bought and read Bardin's book because Richard Bejtlich identified it as a "train wreck," and it was.

(Previously: 2014201320122011201020092008200720062005.)

Monday, November 23, 2015

A few thoughts on OpenBSD 5.8

I've been using OpenBSD since way back at release 2.3 in 1998, so I've gone through upgrades that took a fair amount of work due to incompatible changes, like the switch from ipf to pf for host firewalling or the change to ELF binaries. The upgrade from 5.7 to 5.8 was a pretty smooth and easy one, for the most part. The two most painful changes for me were the replacement of sudo with doas and the dropping of support in the rc.conf for the pf_rules variable.  While sudo is still available as a package, I like the idea of reducing attack surface with a simpler program, so I made the switch. The two things I miss most about sudo are the ability to authenticate for a period of time and the ability to have a single config file across a whole set of servers. The former I'm just living with, the latter I've adjusted to by having a single config file that has lines commented out depending on which server it's on. I did have one moment of concern about the quality of doas when it incorrectly reported the line number on which I had a syntax error in the config file--fortunately, this was just a failure to increment the line count on continuation lines (ending with a "\") which is fixed in the -current release.

The removal of the pf_rules variable support from rc.conf was a bigger issue--I used to just put the default pf.conf rules file in place with each release and upgrade, and keep my changes in a pf.conf.local file that was specified in the pf_rules variable. The effect was that from the period after the upgrade until I noticed the change, my systems were using the default rules and thus more exposed than they were supposed to be. This wasn't the first time an incompatible change decreased my level of security--the removal of tcpwrappers support from SSH was another. I used to use a combination of pf rules and hosts.allow as additional layers of protection on my SSH access, and had a set of tools that allowed me to easily add IP addresses to or remove them from my hosts.allow files. This would have been a layer of defense still in place with the loss of my pf rules, had it still been in existence. Fortunately, I also have SSH on a non-standard port and only allow SSH key logins, not user/password logins, and most of my systems can't be reached on any port without first making a VPN connection, which requires two-factor authentication.

A minor annoying change that was made in 5.8 was putting the file /var/unbound/db/root.key into /etc/changelist, so that the file gets checked daily by the security script. The issue with this is that if you are actually using unbound with DNSSEC, this file changes daily, though only in the comments. My "reportnew" log monitoring tool has a feature that allows you to be notified if files that are expected to change on some periodic schedule do not change, and that would be more appropriate than getting daily notifications that yes, the autotrust anchor file has been updated yet again. But what would really be ideal here would be a check that the non-comment components have not changed. (Others have also complained about this.)

A final issue I've run into with OpenBSD 5.8 is not a new issue, but it's one that still hasn't been fixed with pf. That is that pf logs certain traffic (IGMP in particular) when it matches a rule that does not call for logging. This appears to be the same issue that was fixed earlier this year in pfsense, which is derived from an older fork of pf.

Monday, July 20, 2015

Al Seckel exposed

"I believe that we are rapidly transitioning from an Age of Information to an Age of Misinformation, and in many cases, outright disinformation." -- Al Seckel, in an interview published on Jeffrey Epstein's website, "Jeffrey Epstein Talks Perception with Al Seckel"

Mark Oppenheimer's long-awaited exposé on Al Seckel, "The Illusionist," has now been published and I urge all skeptics to read it. Seckel, the former head of the Southern California Skeptics and a CSICOP Scientific and Technical Consultant who was listed as a "physicist" in every issue of the Skeptical Inquirer from vol. 11, no. 2 (Winter 1987-88) to vol. 15, no. 2 (Winter 1991) despite having no degree in physics, has long been known among skeptical insiders as a person who was misrepresenting himself and taking advantage of others. Most have remained silent over fear of litigation, which Seckel has engaged in successfully in the past.

An example of a legal threat from Seckel is this email he sent to me on May 27, 2014:
Dear Jim,
News has once again reached me that you are acting as Tom McIver's proxy in
spreading misinformation and disinformation about me. Please be aware that
I sued McIver in a Court of Law for Defamation and Slander, and after a
very lengthy discovery process, which involved showing that he fabricated
letters from my old professors (who provided notarized statements that they
did not ever state nor write the letters that McIver circulated, and the
various treasures who were in control of the financial books of the
skeptics, also came forth and testified that no money was taken, and McIver
was unable to prove any of his allegations. The presiding Judge stated that
this was the "worst case of slander and defamation" that he had ever seen.
Nevertheless, even with such a Court Order he is persisting, and using (and
I mean the term "using") you to further propagate erroneous misinformation.
Lately, he has been making his defamatory comments again various people,
and posting links to a news release article by the Courthouse News (a press
release service) that reports the allegations set forth in complaints. Just
because something is "alleged" does not mean it is True. It has to be
proven in a Court of Law. In this case, after a lengthy discovery process
(and I keep excellent records) the opposite of what was alleged was
discovered, and the opposing counsel "amicably" dismissed their charges
against me. The case was officially dismissed. In fact, the opposing
counsel has been active in trying to get the Courthouse News to actively
remove the entire article, and not just add a footnote at the end.
I note that you have been trying to add this link to my wikipedia page. I
have never met you, and am not interested in fighting with you. I am
attaching the official Court document that this case was filed for
dismissal by the opposing counsel. You can verify yourself that this is an
accurate document with the Court. So, once again, McIver has used you.
My attorneys are now preparing a Criminal Complaint against McIver for so
openly violating the Court Order (it is now a criminal offense), and will
once again open the floodgates of a slander and defamation lawsuit against
him and his family, and anyone else, who aids him willing in this process.
This time he will not have his insurance company cover his defense. This
time that axe will come down hard on him.
For now, I will just think you are victim, but please remove any and all
references to me on any of your websites, and that will be the end of it.
You don't want to be caught in the crossfire.
Yours sincerely,
Al Seckel
--
Al Seckel
Cognitive neuroscientist, author, speaker
Contrary to what Seckel writes, we have, in fact, met--I believe it was during the CSICOP conference, April 3-4, 1987, in Pasadena, California.  I am not an agent of Tom McIver, the anthropologist, librarian, and author of the wonderful reference book cataloging anti-evolution materials, Anti-Evolution, who Seckel sued for defamation in 2007, in a case that was settled out of court (see Oppenheimer's article). I have never met Tom McIver, though I hope I will be able to do so someday--he seems to me to be a man of good character, integrity, and honesty.

The news release Seckel mentions is regarding a lawsuit filed by Ensign Consulting Ltd. in 2011 against Seckel charging him with fraud, which is summarized online on the Courthouse News Service website. I wrote a brief account of the case based on that news article on Seckel's Wikipedia page in an edit on March 13, 2011, but it was deleted by another editor in less than an hour.  Seckel is correct that just because something is alleged does not mean that it is true; my summary was clear that these were accusations made in a legal filing.

Seckel and his wife, Isabel Maxwell (daughter of the deceased British-Czech media mogul, Robert Maxwell), rather than fighting the suit or showing up for depositions, filed for bankruptcy.  Ensign filed a motion in their bankruptcy case on December 2, 2011, repeating the fraud allegations.  But as Seckel notes, Ensign did dismiss their case in 2014 prior to his sending me the above email.

So why should anyone care?  Who is Al Seckel, and what was he worried that I might be saying about him? This is mostly answered by the Oppenheimer article, but there is quite a bit more that could be said, and more than what I will say here to complement "The Illusionist."

Al Seckel was the founder and executive director of the Southern California Skeptics, a Los Angeles area skeptics group that met at Caltech.  This was one of the earliest local skeptical groups, with a large membership and prominent scientists on its advisory board.  Seckel has published numerous works including editing two collections of Bertrand Russell's writings for Prometheus Books (both reviewed negatively in the Journal of Bertrand Russell Studies, see here and here).  He has given a TED talk on optical illusions and authored a book with the interesting title, Masters of Deception, which has a forward by Douglas R. Hofstadter.  Seckel was an undergraduate at Cornell University, and developed an association with a couple of cognitive psychology labs at Caltech--in 1998 the New York Times referred to him as a "research associate at the Shimojo Psychophysics Laboratory." His author bios have described him as author of the monthly Neuroquest column at Discover magazine ("About the Author" on Masters of Deception; Seckel has never written that column), as "a physicist and molecular biologist" (first page of Seckel's contribution, "A New Age of Obfuscation and Manipulation" in Robert Basil, editor, Not Necessarily the New Age, 1988, Prometheus Books, pp. 386-395; Seckel is neither a physicist nor a molecular biologist), and, in his TED talk bio, as having left Caltech to continue his work "in spatial imagery with psychology researchers as Harvard" (see Oppenheimer's exchanges with Kosslyn, who has never met or spoken with him and Ganis, who says he has exchanged email with him but not worked with him).

At Cornell, Seckel associated with L. Pearce Williams, a professor of history of science, who had interesting things to say when McIver asked him about their relationship. While in at least one conference bio, Seckel is listed as having been Carl Sagan's teaching assistant, I do not believe that was the case. The Cornell registrar reported in 1991 in response to a query from Pat Linse that Seckel only attended for two semesters and a summer session, though a few places on the web list him as a Cornell alumnus.

Seckel used to hang out at Caltech with Richard Feynman. As the late Helen Tuck, Feyman's administrative assistant, wrote in 1991, Seckel "latched on to Feynman like a leach [sic]." Tuck wrote that she became suspicious of Seckel, and contacted Cornell to find that he did not have a degree from that institution. You can see her full letter, written in response to a query from Tom McIver, here.

As the head of the Southern California Skeptics, Seckel managed to get a column in the Los Angeles Times, titled "Skeptical Eye." Most of his columns were at least partially plagiarized from the work of others, including his column on Sunny the counting dalmation (plagiarized from Robert Sheaffer), his column on tabloid psychics' predictions for 1987 (also plagiarized from Sheaffer), and his column about Martin Reiser's tests of psychic detectives (plagiarized directly from Reiser's work). When Seckel plagiarized Sheaffer, it was brought to the attention of Kent Harker, editor of the Bay Area Skeptics Information Sheet (BASIS), who contacted Seckel about it. Seckel apparently told Harker that Sheaffer had given his permission to allow publication of his work under Seckel's name, which Sheaffer denied when Harker asked. This led to Harker writing to Seckel in 1988 to tell him about Sheaffer's denial, and inform him that he, Seckel, was no longer welcome to reprint any material from BASIS in LASER, the Southern California Skeptics' newsletter. While most skeptical groups gave each other blanket permission to reprint each others' material with attribution, Harker explicitly retracted this permission for Seckel.

This is, I think, a good case study in how the problem of "affiliate fraud"--being taken in by deception by a member of a group you self-identify with--can be possible for skeptics, scientists, and other educated people, just as it is for the more commonly publicized cases of affiliate fraud within religious organizations.

This just scratches the surface of the Seckel story. I hope that those who have been fearful of litigation from Seckel will realize that, given the Oppenheimer story, now is an opportune time for multiple people to come forward and offer each other mutual support that was unhappily unavailable for Tom McIver eight years ago.

(BTW, one apparent error in the Oppenheimer piece--I am unaware of Richard Feynman lending his name for use by a skeptical group. He was never, for example, a CSICOP Fellow, though I'm sure they asked him just as they asked Murray Gell-Mann, who has been listed as a CSICOP Fellow since Skeptical Inquirer vol. 9, no. 3, Spring 1985.)

"Oh, like everyone else, I used to parrot, and on occasion, still do." -- Al Seckel (interview with Jeffrey Epstein)

Corrected 22 July 2015--original mistakenly said Maxwell was Australian.

Update 22 September 2015--an obituary has been published for Al Seckel, stating that he died in France on an unspecified date earlier this year, but there are as yet no online French death records nor French news stories reporting his death. The obituary largely mirrors content put up on alseckel.net, a domain that was registered on September 18 by a user using Perfect Privacy LLC (domaindiscreet.com) to hide their information. (That in itself is not suspicious, it is generally a good practice for individuals who own domain names to protect their privacy with such mechanisms and I do it myself.)

Update 24 September 2015: French police, via the U.S. consulate, confirmed the death of Al Seckel on July 1, 2015. His body was found at the bottom of a cliff in the village of Saint-Cirq-Lapopie.

Update 21 December 2015: A timeline of Al Seckel's activities may be found here.

Update 14 April 2022: Al Seckel's death has been declared a suicide.

Thursday, January 01, 2015

Books read in 2014

Not much blogging going on here lately, but here's my annual list of books read for 2014:
  • James Altucher, The Choose Yourself Stories
  • Nate Anderson, The Internet Police: How Crime Went Online, and the Cops Followed
  • David V. Barrett, A Brief History of Secret Societies: An Unbiased History of Our Desire for Secret Knowledge
  • Peter Burke, A Social History of Knowledge, vol. 2, From the Encyclopedie to Wikipedia
  • Danielle Keats Citron, Hate Crimes in Cyberspace
  • Harry Collins, Are We All Scientific Experts Now?
  • Christopher Hitchens, Hitch 22
  • Christopher Hitchens, Mortality
  • Bruce E. Hunsberger and Bob Altemeyer, Atheists: A Groundbreaking Study of America's Nonbelievers
  • Walter Isaacson, Steve Jobs
  • Brian Krebs, Spam Nation: The Inside Story of Organized Cybercrime--From Global Epidemic to Your Front Door
  • Kembrew McLeod, Pranksters: Making Mischief in the Modern World
  • China Miéville, The City and the City
  • Roger Pielke, Jr., The Climate Fix: What Scientists and Politicians Won't Tell You About Global Warming
  • Michael Sacasas, The Tourist and the Pilgrim: Essays on Life and Technology in the Digital Age
  • Oliver Sacks, Uncle Tungsten: Memories of a Chemical Boyhood
  • James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed
  • Karen Stollznow, God Bless America: Strange and Unusual Religious Beliefs and Practices in the United States
  • Daniel Suarez, Daemon
  • Daniel Suarez, Freedom
  • Nassim Nicholas Taleb, Antifragile
  • Sabrina Verney, XTUL: An Experience of The Process
  • Timothy Wyllie, Love Sex Fear Death: The Inside Story of the Process Church of the Final Judgment
  • Kim Zetter, Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon
I made progress on a few other books (first five this year, next four from last year, last two still not finished from two years ago):
  • Gabriella Coleman, Hacker, Hoaxer, Whistleblower, Spy: The Many Faces of Anonymous
  • Peter Gutmann, Engineering Security
  • Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt
  • Massimo Pigliucci and Maarten Boudry, Philosophy of Pseudoscience: Reconsidering the Demarcation Problem
  • Steven Pinker, The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century
  • Richard Bejtlich, The Practice of Network Security Monitoring
  • Mary Douglas and Aaron Wildavsky, Risk and Culture: An Essay on the Selection of Technological and Environmental Dangers
  • James Grimmelmann, Internet Law: Cases & Problems (v2; v3 is out now)
  • Douglas Hofstadter and Emmanuel Sander, Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
  • Mark Dowd, John McDonald, and Justin Schuh, The Art of Software Security Assessment: Identifying and Avoiding Software Vulnerabilities
  • Michal Zalewski, The Tangled Web: A Guide to Securing Modern Web Applications
Top ten for 2014:  Sacks, Miéville, Isaacson, Hitchens (both), Wyllie, Zetter, Collins, Pielke Jr., Pigliucci and Boudry.

(Previously: 201320122011201020092008200720062005.)