Sendgrid Under Siege from Hacked Accounts

Email service provider Sendgrid is grappling with an unusually large number of customer accounts whose passwords have been cracked, sold to spammers, and abused for sending phishing and email malware attacks. Sendgrid’s parent company Twilio says it is working on a plan to require multi-factor authentication for all of its customers, but that solution may not come fast enough for organizations having trouble dealing with the fallout in the meantime.

Image: Wikipedia

Many companies use Sendgrid to communicate with their customers via email, or else pay marketing firms to do that on their behalf using Sendgrid’s systems. Sendgrid takes steps to validate that new customers are legitimate businesses, and that emails sent through its platform carry the proper digital signatures that other companies can use to validate that the messages have been authorized by its customers.

But this also means when a Sendgrid customer account gets hacked and used to send malware or phishing scams, the threat is particularly acute because a large number of organizations allow email from Sendgrid’s systems to sail through their spam-filtering systems.

To make matters worse, links included in emails sent through Sendgrid are obfuscated (mainly for tracking deliverability and other metrics), so it is not immediately clear to recipients where on the Internet they will be taken when they click.

Dealing with compromised customer accounts is a constant challenge for any organization doing business online today, and certainly Sendgrid is not the only email marketing platform dealing with this problem. But according to multiple emails from readers, recent threads on several anti-spam discussion lists, and interviews with people in the anti-spam community, over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.

Rob McEwen is CEO of Invaluement.com, an anti-spam firm whose data on junk email trends are used to improve the spam-blocking technologies deployed by several Fortune 100 companies. McEwen said no other email service provider has come close to generating the volume of spam that’s been emanating from Sendgrid accounts lately.

“As far as the nasty criminal phishes and viruses, I think there’s not even a close second in terms of how bad it’s been with Sendgrid over the past few months,” he said.

Trying to filter out bad emails coming from a major email provider that so many legitimate companies rely upon to reach their customers can be a dicey business. If you filter the emails too aggressively you end up with an unacceptable number of “false positives,” i.e., benign or even desirable emails that get flagged as spam and sent to the junk folder or blocked altogether.

But McEwen said the incidence of malicious spam coming from Sendgrid has gotten so bad that he recently launched a new anti-spam block list specifically to filter out email from Sendgrid accounts that have been known to be blasting large volumes of junk or malicious email.

“Before I implemented this in my own filtering system a week ago, I was getting three to four phone calls or stern emails a week from angry customers wondering why these malicious emails were getting through to their inboxes,” McEwen said. “And I just am not seeing anything this egregious in terms of viruses and spams from the other email service providers.”

In an interview with KrebsOnSecurity, Sendgrid parent firm Twilio acknowledged the company had recently seen an increase in compromised customer accounts being abused for spam. While Sendgrid does allow customers to use multi-factor authentication (also known as two-factor authentication or 2FA), this protection is not mandatory.

But Twilio Chief Security Officer Steve Pugh said the company is working on changes that would require customers to use some form of 2FA in addition to usernames and passwords.

“Twilio believes that requiring 2FA for customer accounts is the right thing to do, and we’re working towards that end,” Pugh said. “2FA has proven to be a powerful tool in securing communications channels. This is part of the reason we acquired Authy and created a line of account security products and services. Twilio, like other platforms, is forming a plan on how to better secure our customers’ accounts through native technologies such as Authy and additional account level controls to mitigate known attack vectors.”

Requiring customers to use some form of 2FA would go a long way toward neutralizing the underground market for compromised Sendgrid accounts, which are sold by a variety of cybercriminals who specialize in gaining access to accounts by targeting users who re-use the same passwords across multiple websites.

One such individual, who goes by the handle “Kromatix” on several forums, is currently selling access to more than 400 compromised Sendgrid user accounts. The pricing attached to each account is based on volume of email it can send in a given month. Accounts that can send up to 40,000 emails a month go for $15, whereas those capable of blasting 10 million missives a month sell for $400.

“I have a large supply of cracked Sendgrid accounts that can be used to generate an API key which you can then plug into your mailer of choice and send massive amounts of emails with ensured delivery,” Kromatix wrote in an Aug. 23 sales thread. “Sendgrid servers maintain a very good reputation with [email service providers] so your content becomes much more likely to get into the inbox so long as your setup is correct.”

Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.

Single-factor authentication for a company like this in 2020 is just ludicrous given the potential damage and malicious content we’re seeing,” Schwartzman said.

“I understand that it’s a task to invoke 2FA, and given the volume of customers Sendgrid has that’s something to consider because there’s going to be a lot of customer overhead involved,” he continued. “But it’s not like your bank, social media account, email and plenty of other places online don’t already insist on it.”

Schwartzman said if Twilio doesn’t act quickly enough to fix the problem on its end, the major email providers of the world (think Google, Microsoft and Apple) — and their various machine-learning anti-spam algorithms — may do it for them.

“There is a tipping point after which receiving firms start to lose patience and start to more aggressively filter this stuff,” he said. “If seeing a Sendgrid email according to machine learning becomes a sign of abuse, trust me the machines will make the decisions even if the people don’t.”

COVID-19 is driving demand for low-code apps

Now that the great Y Combinator rush is behind us, we’re returning to a topic many of you really seem to care about: no-code and low-code apps and their development.

We’ve explored the theme a few times recently, once from a venture-capital perspective, and another time building from a chat with the CEO of Claris, an Apple subsidiary and an early proponent of low-code work.

Today we’re adding notes from a call with Appian CEO Matt Calkins that took place yesterday shortly after the company released its most recent earnings report.


The Exchange explores startups, markets and money. You can read it every morning on Extra Crunch, or get The Exchange newsletter every Saturday.


Appian is built on low-code development. Having gone public back in 2017, it is the first low-code IPO we can think of. With its Q2 results reported on August 6, we wanted to dig a bit more into what Calkins is seeing in today’s market so we can better understand what is driving demand for low- and no-code development, specifically, and demand for business apps more generally in 2020.

As you can imagine, COVID-19 and the accelerating digital transformation are going to come up in our notes. But, first, let’s take a look at Appian’s quarter quickly before digging into how its low-code-focused CEO sees the world.

Results, expectations

Appian had a pretty good Q2. The company reported $66.8 million in revenue for the three-month period, ahead of market expectations that it would report around $61 million, though collected analyst estimates varied. The low-code platform also beat on per-share profit, reporting a $0.12 per-share loss after adjustments. Analysts had expected a far worse $0.25 per-share deficit.

The period was better than expected, certainly, but it was not a quarter that showed sharp year-over-year growth. There’s a reason for that: Appian is currently shedding professional services revenue (lower-margin, human-powered stuff) for subscription incomes (higher-margin, software-powered stuff). So, as it exchanges one type of revenue for another with total subscription revenue rising a little over 12% in Q2 2020 compared to the year-ago quarter, and professional services revenue falling around 10%, the company’s growth will be slow but the resulting revenue mix improvement is material.

Most importantly, inside of its larger subscription result for the quarter ($41.4 million) were its cloud subscription revenues, worth $29.6 million for the quarter and up 30% compared to the year-ago period. Summing, the company’s least lucrative revenues are falling as its most lucrative accelerate at the fastest clip of any of its cohorts. That’s what you’d want to see if you are an Appian bull.

Shares in the technology company are up around 45% this year. With that, we can get started.

Salesforce confirms it’s laying off around 1,000 people in spite of monster quarter

In what felt like strange timing, Salesforce has confirmed a report in yesterday’s Wall Street Journal that it was laying off around 1,000 people, or approximately 1.9% of the company’s 54,000 strong workforce. This news came in spite of the company reporting a monster quarter on Tuesday, in which it passed $5 billion in quarterly revenue for the first time.

In fact, Wall Street was so thrilled with Salesforce’s results, the company’s stock closed up an astonishing 26% yesterday, adding great wealth to the company’s coffers. It seemed hard to reconcile such amazing financial success with this news.

Yet it was actually something that president and chief financial officer Mark Hawkins telegraphed in Tuesday’s earnings call with industry analysts, although he didn’t come right and use the L (layoff) word. Instead he couched that impending change as a reallocation of resources.

And he talked about strategically shifting investments over the next 12-24 months. “This means we’ll be redirecting some of our resources to fuel growth in areas that are no longer as aligned with the business priority will be now deemphasized,” Hawkins said in the call.

This is precisely how a Salesforce spokesperson put it when asked by TechCrunch to confirm the story. “We’re reallocating resources to position the company for continued growth. This includes continuing to hire and redirecting some employees to fuel our strategic areas, and eliminating some positions that no longer map to our business priorities. For affected employees, we are helping them find the next step in their careers, whether within our company or a new opportunity,” the spokesperson said.

It’s worth noting that earlier this year, Salesforce CEO Marc Benioff pledged there would be no significant layoffs for 90 days.

The 90-day period has long since passed and the company has decided the time is right to make some adjustments to the workforce.

It’s worth contrasting this with the pledge that ServiceNow CEO Bill McDermott made a few weeks after the Benioff tweet, promising not to lay off a single employee for the rest of this year, while also pledging to hire 1,000 people worldwide the remainder of this year, while bringing in 360 summer interns.

How Salesforce beat its own target to reach $20B run rate ahead of schedule

Salesforce launched in 1999, one of the early adherents to what would eventually be called SaaS and cloud computing. On Tuesday, the company reached a huge milestone when it surpassed $5 billion in revenue, putting the SaaS giant on a $20 billion run rate for the first time.

Salesforce revenue has been on a firm upward trajectory for years now, but when the company reached $10 billion in revenue in November 2017, CEO Marc Benioff set the goal for $20 billion right then and there, and five years hence the company beat that goal pretty easily. Here’s what he said at the time:

In fact as the fastest growing enterprise software company ever to reach $10 billion, we are now targeting to grow the company organically to more than $20 billion by fiscal year 2022 and we plan to do that to be the fastest enterprise software company ever to get to $20 billion.

There are lots of elements that have led to that success. As the Salesforce platform evolved, the company has also had an aggressive acquisition strategy, and companies are moving to the cloud faster than ever before. Yet Salesforce has been able to meet that lofty 2017 goal early, while practicing his own unique form of responsible capitalism in the midst of a pandemic.

The platform play

While there are many factors contributing to the company’s revenue growth, one big part of it is the platform. As a platform, it’s not only about providing a set of software tools like CRM, marketing automation and customer service, it’s also giving customers the ability to build solutions to meet their needs on top of that, taking advantage of the work that Salesforce has done to build its own software stack.

Bret Taylor, president and chief operating officer at Salesforce, says the platform has played a huge role in the company’s success. “Actually our platform is behind a huge part of Salesforce’s momentum in multiple ways. One, which is one thing we’ve talked a lot about, is just the technology characteristics of the platform, namely that it’s low code and fast time to value,” he said.

He added, “I would say that these low-code platforms and the ability to stand up solutions quickly is more relevant than ever before because our customers are going to have to respond to changes in their business faster than ever before,” he said.

He pointed to nCino, a company built on top of Salesforce that went public last month as a prime example of this. The company was built on Salesforce, sold in the AppExchange marketplace and provides a way for banking customers to do business online, taking advantage of all that Salesforce has built to do that.

The acquisition strategy

Another big contributing factor to the company’s success is that beyond the core CRM product it brought to the table way back in 1999, it has built a broad set of marketing, sales and service tools and as it has done that, it has acquired many companies along the way to accelerate the product road map.

The biggest of those acquisitions by far was the $15.7 billion Tableau deal, which closed just about a year ago. Taylor sees data fueling the push to digital we are seeing during the pandemic, and Tableau is a key part of that.

“Tableau is so strategic, both from a revenue and also from a technology strategy perspective,” he said. That’s because as companies make the shift to digital, it becomes more important than ever to help them visualize and understand that data in order to understand their customers’ requirements better.

“Fundamentally when you look at what a company needs to do to thrive in an all-digital world, it needs to be able to respond to [rapid] changes, which means creating a culture around that data,” he said. This enables companies to respond more quickly to changes like new customer demands or shifts in the supply chain.

“All of that is about data, and I think the reason why Tableau grew so much this past quarter is that I think that the conversation around data when you’re digitizing your entire company and digitizing the entire economy, data is more strategic than it ever was,” he said.

With that purchase, combined with the $6.5 billion MuleSoft acquisition in 2018, the company feels like it has a way to capture and visualize data wherever it lives in the enterprise. “It’s worth noting how complementary MuleSoft and Tableau are together. I think of MuleSoft as unlocking all your enterprise data, whether it’s on a legacy system or a modern system, and Tableau enables us to understand it, and so it’s a really strategic overall value proposition because we can come up with a really complete solution around data,” Taylor said.

Capitalism with some heart

Benioff was happy to point out in an appearance on Mad Money Tuesday that even as he has made charity and volunteerism a core part of his organization, he has still delivered solid returns for his shareholders. He told Mad Money host Jim Cramer, “This is a victory for stakeholder capitalism. It shows you can do good and do well.” This is a statement he has made frequently in the past to show that you can be a good corporate citizen and give back to your community, while still making money.

Those values are what separates the company from the pack says Paul Greenberg, founder and principal analyst at 56 Group and author of CRM at the Speed of Light. “Salesforce’s genius, and a large part of the reason I don’t expect any serious slowdown in that extraordinary growth, is that they manage to align the technology business with corporate social responsibility in a way that makes them stand out from any other company,” Greenberg told TechCrunch.

Yesterday’s numbers come after Q1 2021, in which the company offered softer guidance as it was giving some of its customers, suffering from the impact of the pandemic, more financial flexibility. As it turns out, that didn’t seem to hurt them, and the guidance for next quarter is looking good too: $5.24 billion to $5.25 billion, up approximately 16% year over year, according to the company.

It’s worth noting that while Benioff pledged no new layoffs for 90 days at the start of the pandemic, with that time now ending, The Wall Street Journal reported yesterday that the company was planning to eliminate 1,000 roles out of the organization’s 54,000 total employees, while giving those workers 60 days to find other roles in the company.

Getting to $20 billion

Certainly getting to that $20 billion run rate is significant, as is the speed with which they were able to achieve that goal, but Taylor sees an evolving company, one that is different than the one it was in 2017 when Benioff set that goal.

“I would say the reason we’ve been able to accelerate is through organic [growth], innovation and acquisitions to really build out this vision of a complete customer [picture]. I think it’s more important than ever before,” he said.

He says that when you look at the way the platform has changed, it’s been about bringing multiple customer experience capabilities together under a single umbrella, and giving customers the tools they need to build these out.

“I think we as a company have constantly redefined what customer relationship management means. It’s not just opportunity management for sales teams. It’s customer service, it’s e-commerce, it’s digital marketing, it’s B2B, it’s B2C. It’s all of the above,” he said.

Box benefits from digital transformation as it raises its growth forecast

Box has always been a bit of an enigma for Wall Street, and perhaps for enterprise software in general. Unlike vendors who shifted to the cloud tools like HR, CRM or ERP, Box has been building a way to manage content in the cloud. It’s been a little harder to understand than these other enterprise software stalwarts, but slowly but surely Box has shifted into a more efficient, and dare we say, profitable public company.

Yesterday the company filed its Q2 2021 earnings report and it was solid. In fact, the company reported revenue of $192.3 million. That’s an increase of 11% year over year and it beat analyst’s expectations of $189.6 million, according to the company. Meanwhile the guidance looked good too, moving from a range of $760 to $768 million for the year to a range of $767 to $770 million.

All of this points to a company that is finding its footing. Let’s not forget, Starboard Value bought a 7.5% stake in the company a year ago, yet the activist investor has mostly stayed quiet and Box seems to be rewarding its patience as the pandemic acts as a forcing function to move customers to the cloud faster — and that seems to be working in Box’s favor.

Let’s get profitable

Box CEO Aaron Levie has not been shy about talking about how the pandemic has pushed companies to move to the cloud much more quickly than they probably would have. He said as a digital company, he was able to move his employees to work from home and remain efficient because of tools like Slack, Zoom, Okta and, yes, Box were in place to help them do that.

All of that helped keep the business going, and even thriving, through the extremely difficult times the pandemic has wrought. “We’re fortunate about how we’ve been able to execute in this environment. It helps that we’re 100% SaaS, and we’ve got a great digital engine to perform the business,” he said.

He added, “And at the same time, as we’ve talked about, we’ve been driving greater profitability. So the efficiency of the businesses has also improved dramatically, and the result was that overall we had a very strong quarter with better growth than expected and better profitability than expected. As a result, we were able to raise our targets on both revenue growth and profitability for the rest of the year,” Levie told TechCrunch.

Let’s get digital

Box is seeing existing customers and new customers alike moving more rapidly to the cloud, and that’s working in its favor. Levie believes that companies are in the process of reassessing their short and longer term digital strategy right now, and looking at what workloads they’ll be moving to the cloud, whether that’s cloud infrastructure, security in the cloud or content.

“Really customers are going to be trying to find a way to be able to shift their most important data and their most important content to the cloud, and that’s what we’re seeing play out within our customer base,” Levie said.

He added, “It’s not really a question anymore if you’re going to go to the cloud, it’s which cloud are you going to go to. And we’ve obviously been very focused on trying to build that leading platform for companies that want to be able to move their data to a cloud environment and be able to manage it securely, drive workflows on it, integrate it across our applications and that’s what we’re seeing,” he said.

That translated into a 60% increase quarter over quarter on the number of large deals over $100,000, and the company crossed 100,000 customers globally on the platform in the most recent quarter, so the approach seems to be working.

Let’s keep building

As with Salesforce a generation earlier, Box decided to build its product set on a platform of services. It enabled customers to tap into these base services like encryption, workflow and metadata and build their own customizations or even fully functional applications by taking advantage of the tools that Box has already built.

Much like Salesforce president and COO Bret Taylor told TechCrunch recently, that platform approach has been an integral part of its success, and Levie sees it similarly for Box. calling it fundamental to his company’s success, as well.

“We would not be here without that platform strategy,” he said. “Because we think about Box as a platform architecture, and we’ve built more and more capabilities into that platform, that’s what is giving us this strategic advantage right now,” he said.

And that hasn’t just worked to help customers using Box, it also helps Box itself to develop new capabilities more rapidly, something that has been absolutely essential during this pandemic when the company has had to react quickly to rapidly changing customer requirements.

Levie is 15 years into his tenure as CEO of Box, but he still sees a company and a market that is just getting started. “The opportunity is only bigger, and it’s more addressable by our product and platform today than it has been at any point in our history. So I think we’re still in the very early stages of digital transformation, and we’re in the earliest stages for how document and content management works in this modern era.”

Confessions of an ID Theft Kingpin, Part II

Yesterday’s piece told the tale of Hieu Minh Ngo, a hacker the U.S. Secret Service described as someone who caused more material financial harm to more Americans than any other convicted cybercriminal. Ngo was recently deported back to his home country after serving more than seven years in prison for running multiple identity theft services. He now says he wants to use his experience to convince other cybercriminals to use their skills for good. Here’s a look at what happened after he got busted.

Hieu Minh Ngo, 29, in a recent photo.

Part I of this series ended with Ngo in handcuffs after disembarking a flight from his native Vietnam to Guam, where he believed he was going to meet another cybercriminal who’d promised to hook him up with the mother of all consumer data caches.

Ngo had been making more than $125,000 a month reselling ill-gotten access to some of the biggest data brokers on the planet. But the Secret Service discovered his various accounts at these data brokers and had them shut down one by one. Ngo became obsessed with restarting his business and maintaining his previous income. By this time, his ID theft services had earned roughly USD $3 million.

As this was going on, Secret Service agents used an intermediary to trick Ngo into thinking he’d trodden on the turf of another cybercriminal. From Part I:

The Secret Service contacted Ngo through an intermediary in the United Kingdom — a known, convicted cybercriminal who agreed to play along. The U.K.-based collaborator told Ngo he had personally shut down Ngo’s access to Experian because he had been there first and Ngo was interfering with his business.

“The U.K. guy told Ngo, ‘Hey, you’re treading on my turf, and I decided to lock you out. But as long as you’re paying a vig through me, your access won’t go away’,” the Secret Service’s Matt O’Neill recalled.

After several months of conversing with his apparent U.K.-based tormentor, Ngo agreed to meet him in Guam to finalize the deal. But immediately after stepping off of the plane in Guam, he was apprehended by Secret Service agents.

“One of the names of his identity theft services was findget[.]me,” O’Neill said. “We took that seriously, and we did like he asked.”

In an interview with KrebsOnSecurity, Ngo said he spent about two months in a Guam jail awaiting transfer to the United States. A month passed before he was allowed a 10 minute phone call to his family and explain what he’d gotten himself into.

“This was a very tough time,” Ngo said. “They were so sad and they were crying a lot.”

First stop on his prosecution tour was New Jersey, where he ultimately pleaded guilty to hacking into MicroBilt, the first of several data brokers whose consumer databases would power different iterations of his identity theft service over the years.

Next came New Hampshire, where another guilty plea forced him to testify in three different trials against identity thieves who had used his services for years. Among them was Lance Ealy, a serial ID thief from Dayton, Ohio who used Ngo’s service to purchase more than 350 “fullz” — a term used to describe a package of everything one would need to steal someone’s identity, including their Social Security number, mother’s maiden name, birth date, address, phone number, email address, bank account information and passwords.

Ealy used Ngo’s service primarily to conduct tax refund fraud with the U.S. Internal Revenue Service (IRS), claiming huge refunds in the names of ID theft victims who first learned of the fraud when they went to file their taxes and found someone else had beat them to it.

Ngo’s cooperation with the government ultimately led to 20 arrests, with a dozen of those defendants lured into the open by O’Neill and other Secret Service agents posing as Ngo.

The Secret Service had difficulty pinning down the exact amount of financial damage inflicted by Ngo’s various ID theft services over the years, primarily because those services only kept records of what customers searched for — not which records they purchased.

But based on the records they did have, the government estimated that Ngo’s service enabled approximately $1.1 billion in new account fraud at banks and retailers throughout the United States, and roughly $64 million in tax refund fraud with the states and the IRS.

“We interviewed a number of Ngo’s customers, who were pretty open about why they were using his services,” O’Neill said. “Many of them told us the same thing: Buying identities was so much better for them than stolen payment card data, because card data could be used once or twice before it was no good to them anymore. But identities could be used over and over again for years.”

O’Neill said he still marvels at the fact that Ngo’s name is practically unknown when compared to the world’s most infamous credit card thieves, some of whom were responsible for stealing hundreds of millions of cards from big box retail merchants.

“I don’t know of anyone who has come close to causing more material harm than Ngo did to the average American,” O’Neill said. “But most people have probably never heard of him.”

Ngo said he wasn’t surprised that his services were responsible for so much financial damage. But he was utterly unprepared to hear about the human toll. Throughout the court proceedings, Ngo sat through story after dreadful story of how his work had ruined the financial lives of people harmed by his services.

“When I was running the service, I didn’t really care because I didn’t know my customers and I didn’t know much about what they were doing with it,” Ngo said. “But during my case, the federal court received like 13,000 letters from victims who complained they lost their houses, jobs, or could no longer afford to buy a home or maintain their financial life because of me. That made me feel really bad, and I realized I’d been a terrible person.”

Even as he bounced from one federal detention facility to the next, Ngo always seemed to encounter ID theft victims wherever he went, including prison guards, healthcare workers and counselors.

“When I was in jail at Beaumont, Texas I talked to one of the correctional officers there who shared with me a story about her friend who lost her identity and then lost everything after that,” Ngo recalled. “Her whole life fell apart. I don’t know if that lady was one of my victims, but that story made me feel sick. I know now that was I was doing was just evil.”

Ngo’s former ID theft service usearching[.]info.

The Vietnamese hacker was released from prison a few months ago, and is now finishing up a mandatory three-week COVID-19 quarantine in a government-run facility near Ho Chi Minh city. In the final months of his detention, Ngo started reading everything he could get his hands on about computer and Internet security, and even authored a lengthy guide written for the average Internet user with advice about how to avoid getting hacked or becoming the victim of identity theft.

Ngo said while he would like to one day get a job working in some cybersecurity role, he’s in no hurry to do so. He’s already had at least one job offer in Vietnam, but he turned it down. He says he’s not ready to work yet, but is looking forward to spending time with his family — and specifically with his dad, who was recently diagnosed with Stage 4 cancer.

Longer term, Ngo says, he wants to mentor young people and help guide them on the right path, and away from cybercrime. He’s been brutally honest about his crimes and the destruction he’s caused. His LinkedIn profile states up front that he’s a convicted cybercriminal.

“I hope my work can help to change the minds of somebody, and if at least one person can change and turn to do good, I’m happy,” Ngo said. “It’s time for me to do something right, to give back to the world, because I know I can do something like this.”

Still, the recidivism rate among cybercriminals tends to be extremely high, and it would be easy for him to slip back into his old ways. After all, few people know as well as he does how best to exploit access to identity data.

O’Neill said he believes Ngo probably will keep his nose clean. But he added that Ngo’s service if it existed today probably would be even more successful and lucrative given the sheer number of scammers involved in using stolen identity data to defraud states and the federal government out of pandemic assistance loans and unemployment insurance benefits.

“It doesn’t appear he’s looking to get back into that life of crime,” O’Neill said. “But I firmly believe the people doing fraudulent small business loans and unemployment claims cut their teeth on his website. He was definitely the new coin of the realm.”

Ngo maintains he has zero interest in doing anything that might send him back to prison.

“Prison is a difficult place, but it gave me time to think about my life and my choices,” he said. “I am committing myself to do good and be better every day. I now know that money is just a part of life. It’s not everything and it can’t bring you true happiness. I hope those cybercriminals out there can learn from my experience. I hope they stop what they are doing and instead use their skills to help make the world better.”

Defeating “Doki” Malware and Container Escapes with Advanced Linux Behavioral Detection

Recently, Intezer cybersecurity researchers uncovered an attack utilizing a new Linux malware targeting publicly accessible Docker servers. The new malware, dubbed “Doki”, is part of an active Ngrok Mining Botnet campaign, primarily targeting exposed Docker servers hosted with popular cloud platforms such as AWS, Azure, and GCP among others. This sophisticated attack exploits misconfigurations in Docker features, which are both common and can be difficult to avoid, and drops the Doki backdoor as one of its payloads.

The initial report noted that “Doki” went unrecognized as malware on VirusTotal for over seven months and claimed it was a “fully undetected backdoor”. Combined with the initial infection’s container escape technique, this has led to fears that enterprises making use of Docker servers are left with little hope of detecting this new kind of attack in the wild, and pressure has naturally mounted on SecOps and DevOps teams to ensure all Docker instances are properly configured in a ‘best effort’ attempt to secure container and cloud workloads. However, while ensuring proper configuration is certainly a fundamental part of an effective security posture, it is also difficult and time consuming; more importantly, it is also not enough to stop attackers that have exploited existing misconfigurations or who go on to discover further container vulnerabilities.

In this post, we show how the container escape and Doki malware attack proceeds, step by step, and demonstrate that neither are “fully undetectable”. We show that this and similar threats can be detected and mitigated against by means of SentinelOne’s Container Escape Protection, part of the SentinelOne Linux and Kubernetes Sentinel Agents.

Container Escape and Privilege Escalation

The main prize for the attackers is to achieve remote code execution on the host, and to this end they leverage the Docker API Create to set up their own containers. As previously reported, by using a legitimate Docker alpine image with curl installed, the attackers are able to use a bind configuration, which internally calls mount syscall, to bind /tmpXXXXXX to the root directory of the hosting server.

Having managed to execute code in the container and get access to the host, the attackers have the option of implementing different persistence methods to overcome the challenge of the average short lifespan of any individual container. In this attack, the initial payload gains persistence in the early stages right after the bind mount configuration by mapping cron to the malicious container.

Detecting the Container Escape with SentinelOne

As Gartner have previously pointed out, enterprises that try to use standard EPP solutions to protect server workloads are putting their business at risk. The only way to detect behaviors that involve correlation between container operations on a host’s file system is through an advanced AI technology that has visibility and understanding of the whole system – both host and containers – at once.

The SentinelOne agent is able to stop this attack precisely because it is constantly monitoring all activities and the malicious cron modifications are immediately detected, as shown below in the console’s threat page. Note how the threat indicators map the activity to MITRE ATT&CK TTPs for the analyst’s convenience:

The console also offers a useful graphical overview of the process tree:

And full logs are readily available showing all events from the current threat within the same interface:

Detecting the “Undetectable” Doki Malware Payload

SentinelOne’s agent is fully able to detect the container escape, but what about the malware that went undiscovered on VirusTotal for so many months and which was said to be “undetectable”?

Certainly, the malware and the initial attack are different steps that attackers could easily use separately; the malware could be dropped from different attack vectors, and it’s equally likely now that Doki has been “discovered” we will see new malware that has yet to be found on VirusTotal or any other malware repository.

Kubernetes Sentinel Agent
Runtime Protection and EDR for Containerized Workloads

Fortunately, the SentinelOne agent does not rely on reputation or cloud connectivity, but analyses processes in real time locally on the device using our advanced machine learning model to detect and protect against abnormal behavior. The on-device agent monitors every process, file and network activity in both the host and containers together, allowing it to capture suspicious and malicious activity autonomously. As the following images show, Doki’s behaviour is immediately recognized by the SentinelOne agent as malicious.

Are There Other Container Escape Techniques?

The particular container escape used in this attack is not the only one available to threat actors. Last year, a security assessment of Kubernetes and Docker presented a different Proof of Concept for achieving a container escape. The PoC relied on another misconfiguration where the container has elevated privileges, either by the --privileged flag or the AppArmor=unconfined flag. The escape can be triggered by an exploit using the Linux cgroups (control groups) mechanism and a ‘release_agent’ file.

Linux control groups are intended to allow multiple Docker containers to run in isolation while limiting and monitoring their use of resources. However, the ‘release_agent’ file contains a command that is executed by the kernel with full privileges on the host once the last task in a cgroup terminates. The PoC abuses this functionality by creating a ‘release_agent’ file with a malicious command, and then killing off all the tasks in the cgroup.

As the cgroup files are present both in the container and on the host, it is possible to modify them from either, which means an attacker can spawn a process inside the cgroup and gain code execution on the host.

# On the host
docker run --rm -it --cap-add=SYS_ADMIN --security-opt apparmor=unconfined ubuntu bash

# In the container
mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/x

echo 1 > /tmp/cgrp/x/notify_on_release
host_path=`sed -n 's/.*perdir=([^,]*).*/1/p' /etc/mtab`
echo "$host_path/cmd" > /tmp/cgrp/release_agent

echo '#!/bin/sh' > /cmd
echo "ps aux > $host_path/output" >> /cmd
chmod a+x /cmd

sh -c "echo $$ > /tmp/cgrp/x/cgroup.procs"

The SentinelOne agent’s Behavioral AI is able to detect this exploitation attempt, providing full visibility and the Storyline of the attack vector that led to this malicious activity.

Conclusion

It is a good strategy for defenders to be familiar with and execute core workload protection strategies, but as the recent Doki and container escape malware attacks show, as soon as there is a weak link in the chain, the attacker will take advantage and such strategies will fail to protect the enterprise.

Modern attack methods in containerized environments in the cloud are gaining traction and becoming increasingly sophisticated. Given the rewards, threat actors are clearly willing to expend more effort to stay under the radar and to defeat “best practices”.

To fully protect your assets, move to a container protection solution, powered by unmatched behavioral AI models, that can autonomously detect and block malware across both hosts and containers. SentinelOne’s server and workload protection is infrastructure agnostic and can be deployed either in containers themselves, or in the machines that host them, in servers or in the cloud. If you would like to see how SentinelOne’s solution can work for you, contact us for more information or request a free demo.


Like this article? Follow us on LinkedIn, Twitter, YouTube or Facebook to see the content we post.

Read more about Cyber Security

Google Cloud Anthos update brings support for on-prem, bare metal

When Google announced Anthos last year at Google Cloud Next, it was a pretty big deal. Here was a cloud company releasing a product that purported to help you move your applications between cloud companies like AWS and Azure — GCP’s competitors — because it’s what customers demanded.

Google tapped into genuine anxiety that tech leaders at customer companies are having over vendor lock-in in the cloud. Back in the client-server days, most of these folks got locked into a tech stack where they were at the mercy of the vendor. It’s something companies desperately want to avoid this go-round.

With Anthos, Google claimed you could take an application, package it in a container and then move it freely between clouds without having to rewrite it for the underlying infrastructure. It was and remains a compelling idea.

This year, the company is updating the product to include a couple of specialty workloads that didn’t get into version 1.0 last year. For starters, many customers aren’t just multi-cloud, meaning they have workloads on various infrastructure cloud vendors, they are also hybrid. That means they still have workloads on-prem in their own data centers, as well as in the cloud, and Google wanted to provide a way to include these workloads in Anthos.

Pali Bhat, VP of product and design at Google Cloud, says they have heard customers still have plenty of applications on premises and they want a way to package them as containerized, cloud-native workloads.

“They do want to be able to bring all of the benefits of cloud to both their own data centers, but also to any cloud they choose to use. And what Anthos enables them to do is go on this journey of modernization and digital transformation and be able to take advantage of it by writing once and running it anywhere, and that’s a really cool vision,” Bhat said.

And while some companies have made the move from on prem to the cloud, they still want the comfort of working on bare metal where they are the only tenant. The cloud typically offers a multi-tenant environment where users share space on servers, but bare metal gives a customer the benefits of being in the cloud with the ability to control their own destiny as they do on prem.

Customers were asking for Anthos to support bare metal, and so Google gave the people what they wanted and are releasing a beta of Anthos for bare metal this week, which Bhat says provides the answer for companies looking to have the benefits of Anthos at the edge.

“[The bare metal support] lets customers run Anthos […] at edge locations without using any hypervisor. So this is a huge benefit for customers who are looking to minimize unnecessary overhead and unlock new use cases, especially both in the cloud and on the edge,” Bhat said.

Anthos is part of a broader cloud modernization platform that Google Cloud is offering customers that includes GKE (the Kubernetes engine), Cloud Functions (the serverless offering) and Cloud Run (container run time platform). Bhat says this set of products taps into a couple of trends they are seeing with customers. First of all, as we move deeper into the pandemic, companies are looking for ways to cut costs while making a faster push to the cloud. The second is taking advantage of that push by becoming more agile and innovative.

It seems to be working. Bhat reports that in Q2, the company has seen a lot of interest. “One of the things in Q2 of 2020 that we’ve seen is that just Q2, over 100,000 companies used our application modernization platform and services,” he said.

MIT CSAIL grad launches machine learning platform with $10M Series A

Manasi Vartak, founder and CEO of Verta, conceived of the idea of the open-source project ModelDB database as a way to track versions of machine models while she was still in grad school at MIT. After she graduated, she decided to expand on that vision to build a product that could not only track model versions, but provide a way to operationalize them — and Verta was born.

Today, that company emerged from stealth with a $10 million Series A led by Intel Capital with participation from General Catalyst, which also led the company’s $1.7 million seed round.

Beyond providing a place to track model versioning, which ModelDB gave users, Vartak wanted to build a platform for data scientists to deploy those models into production, which has been difficult to do for many companies. She also wanted to make sure that once in production, they were still accurately reflecting the current data and not working with yesterday’s playbook.

“Verta can track if models are still valid and send out alarms when model performance changes unexpectedly,” the company explained.

Verta interface

Image Credits: Verta

Vartak says having that open-source project helped sell the company to investors early on, and acts as a way to attract possible customers now. “So for our seed round, it was definitely different because I was raising as a solo founder, a first-time founder right out of school, and that’s where having the open-source project was a huge win,” she said.

Certainly Mark Rostick, VP and senior managing director at lead investor Intel Capital, recognized that Verta was trying to solve a fundamental problem around machine learning model production. “Verta is addressing one of the key challenges companies face when adopting AI — bridging the gap between data scientists and developers to accelerate the deployment of machine learning models,” Rostick said.

While Vartak wasn’t ready to talk about how many customers she has just yet at this early stage of the company, she did say there were companies using the platform and getting models into production much faster.

Today, the company has 9 employees, and even at this early stage, she is taking diversity very seriously. In fact, her current employee makeup includes four Indian, three Caucasian, one Latino and one Asian, for a highly diverse mix. Her goal is to continue on this path as she builds the company. She is looking at getting to 15 employees this year, then doubling that by next year.

One thing Vartak also wants to do is have a 50/50 gender split, something she was able to achieve while at MIT in her various projects, and she wants to carry on with her company. She is also working with a third party, Sweat Equity Ventures, to help with recruiting diverse candidates.

She says that she likes to work iteratively to build the platform, while experimenting with new features, even with her small team. Right now, that involves interoperability with different machine learning tools out there like Amazon SageMaker or Kubeflow, the open-source machine learning pipeline tool.

“We realized that we need to meet customers where they are at their level of maturity. So we focused a lot the last couple of quarters on building a system that was interoperable so you can pick and choose the components kind of like Lego blocks and have a system that works end to end seamlessly.”

Cisco acquiring BabbleLabs to filter out the lawn mower screeching during your video conference

We’ve all been in a video conference, especially this year, when the neighbor started mowing the lawn or kids were playing outside your window — and it can get pretty loud. Cisco, which owns the WebEx video conferencing service, wants to do something about that, and late yesterday it announced it was going to acquire BabbleLabs, a startup that can help filter out background noise.

BabbleLabs has a very particular set of skills. It uses artificial intelligence to enhance the speaking voice, while filtering out those unwanted background noises that seem to occur whenever you happen to be in a meeting.

Interestingly enough, Cisco also sees this as a kind of privacy play by removing background conversation. Jeetu Patel, senior vice president and general manager in the Cisco Security and Applications Business Unit, says that this should go a long way toward improving the meeting experience for Cisco users.

“Their technology is going to provide our customers with yet another important innovation — automatically removing unwanted noise — to continue enabling exceptional Webex meeting experiences,” Patel, who was at Box for many years before joining Cisco, recently said in a statement.

In a blog post, BabbleLabs CEO and co-founder Chris Rowen wrote that conversations about being acquired by Cisco began just recently, and the deal came together pretty quickly. “We quickly reached a common view that merging BabbleLabs into the Cisco Collaboration team could accelerate our common vision dramatically,” he wrote.

BabbleLabs, which launched three years ago and raised $18 million, according to Crunchbase, had an interesting, but highly technical idea. That can sometimes be difficult to translate into a viable commercial product, but makes a highly attractive acquisition target for a company like Cisco.

Brent Leary, founder and principal analyst at CRM Essentials, says this acquisition could be seen as part of a broader industry consolidation. “We’re seeing consolidation taking place as the big web conferencing players are snapping up smaller players to round out their platforms,” he said.

He added, “WebEx may not be getting the attention that Zoom is, but it still has a significant presence in the enterprise, and this acquisition will allow them to keep improving their offering.”

The deal is expected to close in the current quarter after regulatory approval. Upon closing, BabbleLabs employees will become part of Cisco’s Collaboration Group.