Microsoft Teams has been down this morning

Microsoft Teams, the collaboration platform that competes with Slack, has been down since about 8:30 am ET. Microsoft reports the outage was due to an expired certificate.

Microsoft first posted that an outage was in progress on its Office 365 Status Twitter feed about 9:00 am ET, stating the company was looking into the problem.

At approximately 10:00 am ET, the company posted the reason for the problem, an expired certificate, which frankly, has to be pretty embarrassing for the group responsible for keeping the Teams service running.

About an hour ago, the company updated the status again, indicating it had begun deploying the updated certificate.

Some customers have begun reporting on Twitter that service has been restored.

Microsoft has kept the status updates pretty business like, but has not apologized to its 20 million users as of publication. The company is in the midst of a battle for hearts and minds in the enterprise collaboration space with Slack, and a preventable outage has to be awkward for them.

The company will no doubt do a post-mortem to figure out how this mistake happened and how to prevent this kind of issue from taking down the site again. While every service is going to experience an outage from time-to-time, it’s up to the organization to understand why it happened and put systems in place to keep a preventable incident like this one from happening again in the future.

Ginni Rometty leaves complex legacy as she steps away as IBM CEO

When Ginni Rometty steps down as CEO at IBM in April and her replacement Arvind Krishna takes the helm, more than eight years will have passed since she took the reins at Big Blue. The executive helped lead a massive transformation, but IBM has had a bumpy financial ride throughout her tenure — at one time recording an astonishing 22 straight quarters of declining revenue.

To be fair, Rometty took over at a tumultuous time when technology was shifting from on-prem software stacks to the cloud. She saw what was coming and used the company’s considerable cash position to buy what she needed to make that switch while taking advantage of IBM’s extensive R&D to build other pieces in-house. But the transition took time, which resulted in some financial missteps.

She deserves credit for trying to move the battleship in a new direction — culminating with the $34 billion purchase of Red Hat — even if the results were ultimately mixed.

Leading the way

Rometty was the first woman to lead IBM in an industry where female CEOs are scarce. When she came on board in 2012, there were just 21 women running Fortune 500 companies; last year, that number had risen to 33, still a paltry 6.6%. Along with Safra Catz at Oracle and Lisa Su of Advanced Micro Devices, Rometty has been part of a small group of female CEOs at large technology companies.

What Nutanix got right (and wrong) in its IPO roadshow

Back in 2016, Nutanix decided to take the big step of going public. Part of that process was creating a pitch deck and presenting it during its roadshow, a coming-out party when a company goes on tour prior to its IPO and pitches itself to investors of all stripes.

It’s a huge moment in the life of any company, and after talking to CEO Dheeraj Pandey and CFO Duston Williams, one we better understood. They spoke about how every detail helped define their company and demonstrate its long-term investment value to investors who might not have been entirely familiar with the startup or its technology.

Pandey and Williams reported going through more than 100 versions of the deck before they finished the one they took on the road. Pandey said they had a data room checking every fact, every number — which they then checked yet again.

In a separate Extra Crunch post, we looked at the process of building that deck. Today, we’re looking more closely at the content of the deck itself, especially the numbers Nutanix presented to the world. We want to see what investors did more than three years ago and what’s happened since — did the company live up to its promises?

Plan of attack

Iowa Prosecutors Drop Charges Against Men Hired to Test Their Security

On Sept. 11, 2019, two security experts at a company that had been hired by the state of Iowa to test the physical and network security of its judicial system were arrested while probing the security of an Iowa county courthouse, jailed in orange jumpsuits, charged with burglary, and held on $100,000 bail. On Thursday Jan. 30, prosecutors in Iowa announced they had dropped the criminal charges. The news came while KrebsOnSecurity was conducting a video interview with the two accused (featured below).

The courthouse in Dallas County, Iowa. Image: Wikipedia.

Gary DeMercurio, 43 of Seattle, and Justin Wynn, 29 of Naples, Fla., are both professional penetration testers employed by Coalfire Labs, a security firm based in Westminster, Colo. Iowa’s State Court Administration had hired the company to test the security of its judicial buildings.

Under the terms of their contract (PDF), DeMercurio and Wynn were permitted to impersonate staff and contractors, provide false pretenses to gain physical access to facilities, “tailgate” employees into buildings, and access restricted areas of those facilities. The contract said the men could not attempt to subvert alarm systems, force-open doors, or access areas that require protective equipment.

When the duo’s early-morning Sept. 11 test of the security at the courthouse in Dallas County, Iowa set off an audible security alarm, they followed procedure and waited on-site for the police. DeMercurio and Wynn said when the county’s sheriff deputies arrived on the scene just a few minutes later, they told the officers who they were and why they were there, and that they’d obtained entry to the premises via an unlocked door.

“They said they found a courthouse door unlocked, so they closed it from the outside and let it lock,” Dan Goodin of Ars Technica wrote of the ordeal in November. “Then they slipped a plastic cutting board through a crack in the door and manipulated its locking mechanism. (Pentesters frequently use makeshift or self-created tools in their craft to flip latches, trigger motion-detected mechanisms, and test other security systems.) The deputies seemed impressed.”

To assuage concerns they might be burglars, DeMercurio and Wynn produced an authorization letter detailing the job they’d been hired to do and listing the names and mobile phone numbers of Iowa state employees who could verify their story.

After contacting some of the court officials listed in the letter, the deputies seemed satisfied that the men weren’t thieves. That is, until Dallas County Sheriff Chad Leonard showed up.

“The pentesters had already said they used a tool to open the front door,” Goodin recounted. “Leonard took that to mean the men had violated the restriction against forcing doors open. Leonard also said the men attempted to turn off the alarm—something Coalfire officials vehemently deny. In Leonard’s mind that was a second violation. Another reason for doubt: one of the people listed as a contact on the get-out-of-jail-free letter didn’t answer the deputies’ calls, while another said he didn’t believe the men had permission to conduct physical intrusions.”

DeMercurio and Wynn were arrested, jailed, and held for nearly 24 hours before being released on a $100,000 bail. Initially they were charged with felony third-degree burglary and possessing burglary tools, although those charges were later downgraded to misdemeanor trespass.

What initially seemed to Coalfire as a momentary lapse of judgment by Iowa authorities quickly morphed into the surreal when state lawmakers held hearings questioning why and how someone in the state’s employ could have so recklessly endangered the safety and security of its citizens.

DeMercurio and Wynn, minus the orange jumpsuits.

Judicial Branch officials in Dallas County said in response to this grilling that they didn’t expect Coalfire’s physical penetration testing to be conducted outside of business hours. State Sen. Amy Sinclair was quoted as telling her colleagues that “the hiring of an outside company to break into the courthouses in September created ‘significant danger, not only to the contractors, but to local law enforcement, and members of the public.’”

“Essentially a branch of government has contracted with a company to commit crimes, and that’s very troubling,” lamented Iowa state Sen. Zach Whiting. “I want to find out who needs to be held accountable for this and how we can do that.”

Those strong words clashed with a joint statement released Thursday by Coalfire and Dallas County Attorney Charles Sinnard:

“Ultimately, the long-term interests of justice and protection of the public are not best served by continued prosecution of the trespass charges,” the statement reads. “Those interests are best served by all the parties working together to ensure that there is clear communication on the actions to be taken to secure the sensitive information maintained by the judicial branch, without endangering the life or property of the citizens of Iowa, law enforcement or the persons carrying out the testing.

Matthew Linholm, an attorney representing DeMercurio and Wynn in the case, said the justice system ceases to serve its crucial function and loses credibility when criminal accusations are used to advance personal or political agendas.

“Such a practice endangers the effective administration of justice and our confidence in the criminal justice system,” Linholm told The Des Moines Register, which broke the news of the dropped charges.

While the case against Coalfire’s employees has rallied many in the cybersecurity community around the accused, not everyone sees this dispute in black-and-white. Chris Nickerson, a digital intrusion specialist and founder of LARES Consulting, said in a Twitter post Thursday that “when a company puts us in harm’s way due to their poor planning, failed sales education, inadequate project management and deplorable contract management…We shouldn’t celebrate them. We should hold them accountable.”

Asked to elaborate, Nickerson referred to a recent podcast which touched on the arrests.

“The things that concern me about this situation are more of the pieces of safety that exist across how the industry instruments doing these types of engagements,” Nickerson said. “They seem very, very reasonable and obvious once they become obvious but until then they’re completely foreign to people.”

“It’s really on the owners of the organization to educate the customer of those potential pitfalls,” Nickerson continued. “Because there isn’t a good standard. We haven’t all gotten together and institutionalized the knowledge that we have in our heads and dump it down to paper so that someone who is new to the field being tasked with this can go through and say, ‘Hey, did you ask them if the city versus the state versus the building owner and the real estate people…are all of these people in lock step?’”

Coalfire CEO Tom McAndrew seemed to address this point in our interview Thursday, saying there were two unique aspects of this particular engagement. First, although the client in this case said they did not want Coalfire to make local law enforcement aware of the ongoing engagement prior to testing the physical security of the site, it was clear after the fact that state officials never did that on their own.

More importantly, McAndrew said, there was ambiguity around who actually owned the buildings that they were hired to test.

“If you’re doing a test for the state and you walk into the building and it’s the courthouse and you’re doing a test for the court system, you’d think that they would have jurisdiction or own it, and that turned out not to be the case in this scenario because there’s some things the state owns and some things the county owns, and that was something we weren’t aware of as we did some of this work,” he said. “We didn’t understand the nuances.”

Asked what Coalfire has learned from this ordeal, McAndrew said his company is likely to insist that local, state and even federal law enforcement be informed in advance of any penetration tests, at least as far as those engagements relate to public entities.

“When we look at the contracts and we look at who’s authorized to do what…typically, if a [chief security officer] says test these IP addresses, we would say okay that’s enough,” he said. “But we’re questioning from a legal perspective at what point does that need to have legal counsel review.”

McAndrew said it’s probably time for experts from various corners of the pen testing community to collaborate in documenting best practices that might help others avoid a repeat of the scenario in Dallas County.

“There’s no standard in the industry,” he said. “When it comes to these sorts of issues in red teaming — the legal challenges and the contracts — there’s really nothing out there. There are some things that can’t be undone. There’s the mugshots that are out there forever, but even as we get the charges dropped, these are permanently going to be in the federal database. This is a permanent thing that will reside with them and there’s no legal way we’re aware of to get these charges removed from the federal database.”

McAndrew said while he remains frustrated that it took so long to resolve this dispute, he doesn’t believe anyone involved acted with malicious intent.

“I don’t think there were any bad people,” he said. “Everyone was trying to do the right things — from law enforcement to the sheriff to the judges to the county — they all had the right intentions. But they didn’t necessarily all have the right information, and possibly people made decisions at levels they weren’t really authorized to do. Normally that’s not really our call, but I think people need to be thinking about that.”