Notes on Cruise's pedestrian accident | Patreon

This is a set of notes on the Quinn Emanuel report on Cruise's handling of the 2023-10-02 accident where a Cruise autonomous vehicle (AV) hit a pedestrian, stopped, and then started moving again with the pedestrian stuck under the bottom of the AV, dragging the pedestrian 20 feet. After seeing some comments about this report, I read five stories on this report and then skimmed the report and my feeling is that the authors of four of the stories probably didn't read the report, and that people who were commenting had generally read stories by journalists who did not appear to read the source material, so the comments were generally way off base. As we previously discussed, it's common for summaries to be wildly wrong, even when they're summarizing a short paper that's easily read by laypeople, so of course summaries of a 200-page report are likely to be misleading at best.

On reading the entire report, I'd say that Cruise both looks better and worse than in the articles I saw, which is the same pattern we saw when we looked at the actual source for Exhibits H and J from Twitter v. Musk, the United States v. Microsoft Corp. docs, etc.; just as some journalists seem to be pro/anti-Elon Musk and pro/anti-Microsoft, willing to push an inaccurate narrative to dunk on them to the maximum extent possible or exonerate them to the maximum extent possible, we see the same thing here with Cruise. And as we saw in those cases, despite some articles seemingly trying to paint Cruise in the best or worst light possible, the report itself has material that is more positive and more negative than we see in the most positive or negative stories.

Aside from correcting misleading opinions on the report, I find the report interesting because it's rare to see any kind of investigation over what went wrong in tech in this level of detail, let alone a public one. We often see this kind of investigation in safety critical systems and sometimes see in sports as well as for historical events, but tech events are usually not covered like this. Of course companies do post-mortems of incidents, but you generally won't see a 200-page report on a single incident, nor will the focus of post-mortems be what the focus was here. In the past, we've noted that a lot can be learned by looking at the literature and incident reports on safety-critical systems, so of course this is true here as well, where we see a safety-critical system that's more tech adjacent than the ones we've looked at previously.

The length and depth of the report here reflects a difference in culture between safety-critical systems and "tech". The behavior that's described as unconscionable in the report is not only normal in tech, but probably more transparent and above board than you'd see at most major tech companies; I find the culture clash between tech and safety-critical systems interesting as well. I attempted to inject as little of my opinion as possible into the report as possible, even in cases where knowledge of tech companies or engineering meant that I would've personally written something different. For more opinions, see the section at the end.

REPORT TO THE BOARDS OF DIRECTORS OF CRUISE LLC, GM CRUISE HOLDINGS LLC, AND GENERAL MOTORS HOLDINGS LLC REGARDING THE OCTOBER 2, 2023 ACCIDENT IN SAN FRANCISCO

I. Introduction

A. Overview

B. Scope of Review

C. Review Plan Methodology and Limitations

D. Summary of Principal Findings and Conclusions

II. THE FACTS REGARDING THE OCTOBER 2 ACCIDENT

A. Background Regarding Cruise’s Business Operations

B. Key Facts Regarding the Accident

C. Timeline of Key Events

D. Video Footage of the Accident

E. The Facts Regarding What Cruise Knew and When About the October 2 Accident

1. Facts Cruise Learned the Evening of October 2

a. Accident Scene
b. Virtual "Sev-0 War Room"
c. Initial Media Narrative About the October 2 Accident

2. Facts Cruise Learned on October 3

a. The 12:15 a.m. "Sev-0 Collision SFO" Meeting
b. Engineer’s 3:45 a.m. Slack Message
c. The 6:00 a.m. Crisis Management Team (CMT) Meeting
d. The 6:45 a.m. Senior Leadership Team (SLT) Meeting
e. The 7:45 a.m. and 10:35 a.m. Engineering and Safety Team

Meetings

f. The 12:05 p.m. CMT Meeting
g. The 12:40 p.m. SLT Meeting
h. The 6:05 p.m. CMT Meeting

3. Cruise’s Response to the Forbes Article

III. CRUISE’S COMMUNICATIONS WITH REGULATORS, CITY OFFICIALS, AND OTHER STAKEHOLDERS

A. Overview of Cruise’s Initial Outreach and Meetings with Regulators

B. The Mayor’s Office Meeting on October 3

C. Cruise’s Disclosures to the National Highway Traffic Safety Administration (NHTSA)

1. Cruise’s Initial Outreach on October 3

2. Cruise’s NHTSA Pre-Meeting

3. Cruise’s Meeting with NHTSA on October 3

4. Cruise’s NHTSA Post-Meeting on October 3

5. Cruise’s Interactions with NHTSA on October 12, 13, and 16

a. October 12 Call
b. October 13 Meeting
c. October 16 PE

6. Cruise’s NHTSA Reports Regarding the October 2 Accident

a. NHTSA 1-Day Report
b. NHTSA 10-Day Report
c. NHTSA 30-Day Report

7. Conclusions Regarding Cruise’s Interactions with NHTSA

D. Cruise’s Disclosures to the Department of Motor Vehicles (DMV)

1. Cruise’s Initial Outreach to the DMV and Internal Discussion of Which Video to Show

2. DMV’s Response to Cruise’s Outreach

3. Cruise’s DMV Pre-Meeting

4. Cruise’s October 3 Meeting with the DMV

a. DMV Meeting Discussions
b. Cruise’s Post-DMV Meeting Reflections

5. Cruise’s October 10 Communications with DMV

6. Cruise’s October 11 Meeting with the DMV

7. Cruise’s October 13 Meeting with the DMV

8. Cruise’s October 16 Meeting with the DMV

9. Cruise’s October 23 Communications with the DMV

10. DMV’s October 24 Suspension Order

11. Post-October 24 DMV Communications

12. Conclusions Regarding Cruise’s Communications with the DMV

E. Cruise’s Disclosures to the SF MTA, SF Fire Department, and SF Police

Department

F. Cruise’s Disclosures to the California Public Utilities Commission (CPUC)

1. Cruise’s October 3 Communications with the CPUC

2. CPUC’s October 5 Data Request

3. Cruise’s October 19 Response to CPUC’s Data Request

4. Conclusions Regarding Cruise’s Disclosures to the CPUC

G. Cruise’s Disclosures to Other Federal Officials

IV. THE AFTERMATH OF THE OCTOBER 2 ACCIDENT

A. The Cruise License Suspension by the DMV in California

B. The NHTSA PE Investigation and Safety Recall

C. The CPUC’s “Show Cause Ruling”

D. New Senior Management of Cruise and the Downsizing of Cruise

V. SUMMARY OF FINDINGS AND CONCLUSIONS

VI. RECOMMENDATIONS

Appendix

back to danluu.com

I don't have much to add to this. I certainly have opinions, but I don't work in automotive and haven't dug into it enough to feel informed enough to add my own thoughts. In one discussion I had with a retired exec who used to work on autonomous vehicles, on incident management at Cruise vs. tech companies Twitter or Slack, the former exec said:

You get good at incidents given a steady stream of incidents of varying severity if you have to handle the many small ones. You get terrible at incidents if you can cover up the small ones until a big one happens. So it's not only funny but natural for internet companies to do it better than AV companies I think

On the "minimal risk condition" pullover maneuver, this exec said:

These pullover maneuvers are magic pixie dust making AVs safe: if something happens, we'll do a safety pullover maneuver

And on the now-deleted blog post, "A detailed review of the recent SF hit-and-run incident", the exec said:

Their mentioning of regulatory ADAS test cases does not inspire confidence; these tests are shit. But it's a bit unfair on my part since of course they would mention these tests, it doesn't mean they don't have better ones

On how regulations and processes making safety-critical industries safer and what you'd do if you cared about safety vs. the recommendations in the report, this exec said

[Dan,] you care about things being done right. People in these industries care about compliance. Anything "above the state of the art" buys you zero brownie points. eg for [X], any [Y] ATM are not required at all. [We] are better at [X] than most and it does nothing for compliance ... OTOH if a terrible tool or process exists that does nothing good but is considered "the state of the art" / is mandated by a standard, you sure as hell are going to use it

If you're looking for work, Freshpaint is hiring a recruiter, Software Engineers, and a Support Engineer. I'm in an investor, so you should consider my potential bias, but they seem to have found product-market fit and are growing extremely quickly (revenue-wise)

Thanks to an anonymous former AV exec, Justin Blank, and 5d22b for comments/corrections/discussion.

Appendix: a physical hardware curiosity

One question I had for the exec mentioned above, which wasn't relevant to this case, but is something I've wondered about for a while, is why the AVs that I see driving don't have upgraded tires and brakes. You can get much shorter stopping distances from cars that aren't super heavy by upgrading their tires and brakes, but the AVs I've seen have not had this done.

In this case, we can't do the exact comparison from an upgraded vehicle to the base vehicle because the vehicle dynamics data was redacted from section 3.3.3, table 9, and figure 40 of the appendix, but it's common knowledge that the simplest safety upgrade you can make on a car is upgrading the tires (and, if relevant, the brakes). One could argue that this isn't worth the extra running cost, or the effort (for the low-performance cars that I tend to see converted into AVs, getting stopping distances equivalent to a sporty vehicle would generally require modifying the wheel well so that wider tires don't rub) but, as an outsider, I'd be curious to know what the cost benefit trade-off on shorter stopping distances is.

They hadn't considered it before, but thought that better tires and brake would make a difference in a lot of other cases and prevent accients and explained the lack of this upgrade by:

I think if you have a combination of "we want to base AV on commodity cars" and "I am an algorithms guy" mindset you will not go look at what the car should be.

And, to be clear, upgraded tires and brakes would not have changed the outcome in this case. The timeline from the Exponent report has

Looking at actual accelerometer data from a car with upgraded tires and brakes, stopping time from 19.1mph for that car was around 0.8s, so this wouldn't have made much difference in this case. If brakes aren't pre-charged before attempting to brake, there's significant latency when initially braking, such that 0.25s isn't enough for almost any braking to have occurred, which we can see from the speed only being 0.5mph slower in this case.

Another comment from the exec is that, while a human might react to the collision at -2.9s and slow down or stop, "scene understanding" as a human might do it is non-existent in most or perhaps all AVs, so it's unsurprising that the AV doesn't react until the pedestrian is in the AV's path, whereas a human, if they noticed the accident in the adjacent lane, would likely drastically slow down or stop (the exec guessed that most humans would come to a complete stop, whereas I guessed that most humans would slow down). The exec was also not surprised by the 530ms latency between the pedestrian landing in the AV's path and the AV starting to attempt to apply the brakes although, as a lay person, I found 530ms surprising.

On the advantage of AVs and ADAS, as implemented today, compared to a human who's looking in the right place, paying attention, etc., the exec said

They mainly never get tired or drink and hopefully also run in that terrible driver's car in the next lane. For [current systems], it's reliability and not peak performance that makes it useful. Peak performance is definitely not superhuman but subhuman