The five technical challenges Cerebras overcame in building the first trillion-transistor chip

Superlatives abound at Cerebras, the until-today stealthy next-generation silicon chip company looking to make training a deep learning model as quick as buying toothpaste from Amazon. Launching after almost three years of quiet development, Cerebras introduced its new chip today — and it is a doozy. The “Wafer Scale Engine” is 1.2 trillion transistors (the most ever), 46,225 square millimeters (the largest ever), and includes 18 gigabytes of on-chip memory (the most of any chip on the market today) and 400,000 processing cores (guess the superlative).

CS Wafer Keyboard Comparison

Cerebras’ Wafer Scale Engine is larger than a typical Mac keyboard (via Cerebras Systems).

It’s made a big splash here at Stanford University at the Hot Chips conference, one of the silicon industry’s big confabs for product introductions and roadmaps, with various levels of oohs and aahs among attendees. You can read more about the chip from Tiernan Ray at Fortune and read the white paper from Cerebras itself.

Superlatives aside though, the technical challenges that Cerebras had to overcome to reach this milestone I think is the more interesting story here. I sat down with founder and CEO Andrew Feldman this afternoon to discuss what his 173 engineers have been building quietly just down the street here these past few years, with $112 million in venture capital funding from Benchmark and others.

Going big means nothing but challenges

First, a quick background on how the chips that power your phones and computers get made. Fabs like TSMC take standard-sized silicon wafers and divide them into individual chips by using light to etch the transistors into the chip. Wafers are circles and chips are squares, and so there is some basic geometry involved in subdividing that circle into a clear array of individual chips.

One big challenge in this lithography process is that errors can creep into the manufacturing process, requiring extensive testing to verify quality and forcing fabs to throw away poorly performing chips. The smaller and more compact the chip, the less likely any individual chip will be inoperative, and the higher the yield for the fab. Higher yield equals higher profits.

Cerebras throws out the idea of etching a bunch of individual chips onto a single wafer in lieu of just using the whole wafer itself as one gigantic chip. That allows all of those individual cores to connect with one another directly — vastly speeding up the critical feedback loops used in deep learning algorithms — but comes at the cost of huge manufacturing and design challenges to create and manage these chips.

CS Wafer Sean

Cerebras’ technical architecture and design was led by co-founder Sean Lie. Feldman and Lie worked together on a previous startup called SeaMicro, which sold to AMD in 2012 for $334 million (via Cerebras Systems).

The first challenge the team ran into, according to Feldman, was handling communication across the “scribe lines.” While Cerebras’ chip encompasses a full wafer, today’s lithography equipment still has to act like there are individual chips being etched into the silicon wafer. So the company had to invent new techniques to allow each of those individual chips to communicate with each other across the whole wafer. Working with TSMC, they not only invented new channels for communication, but also had to write new software to handle chips with trillion-plus transistors.

The second challenge was yield. With a chip covering an entire silicon wafer, a single imperfection in the etching of that wafer could render the entire chip inoperative. This has been the block for decades on whole-wafer technology: due to the laws of physics, it is essentially impossible to etch a trillion transistors with perfect accuracy repeatedly.

Cerebras approached the problem using redundancy by adding extra cores throughout the chip that would be used as backup in the event that an error appeared in that core’s neighborhood on the wafer. “You have to hold only 1%, 1.5% of these guys aside,” Feldman explained to me. Leaving extra cores allows the chip to essentially self-heal, routing around the lithography error and making a whole-wafer silicon chip viable.

Entering uncharted territory in chip design

Those first two challenges — communicating across the scribe lines between chips and handling yield — have flummoxed chip designers studying whole-wafer chips for decades. But they were known problems, and Feldman said that they were actually easier to solve than expected by re-approaching them using modern tools.

He likens the challenge to climbing Mount Everest. “It’s like the first set of guys failed to climb Mount Everest, they said, ‘Shit, that first part is really hard.’ And then the next set came along and said ‘That shit was nothing. That last hundred yards, that’s a problem.’ ”

And indeed, the toughest challenges, according to Feldman, for Cerebras were the next three, since no other chip designer had gotten past the scribe line communication and yield challenges to actually find what happened next.

The third challenge Cerebras confronted was handling thermal expansion. Chips get extremely hot in operation, but different materials expand at different rates. That means the connectors tethering a chip to its motherboard also need to thermally expand at precisely the same rate, lest cracks develop between the two.

As Feldman explained, “How do you get a connector that can withstand [that]? Nobody had ever done that before, [and so] we had to invent a material. So we have PhDs in material science, [and] we had to invent a material that could absorb some of that difference.”

Once a chip is manufactured, it needs to be tested and packaged for shipment to original equipment manufacturers (OEMs) who add the chips into the products used by end customers (whether data centers or consumer laptops). There is a challenge though: Absolutely nothing on the market is designed to handle a whole-wafer chip.

CS Wafer Inspection

Cerebras designed its own testing and packaging system to handle its chip (via Cerebras Systems).

“How on earth do you package it? Well, the answer is you invent a lot of shit. That is the truth. Nobody had a printed circuit board this size. Nobody had connectors. Nobody had a cold plate. Nobody had tools. Nobody had tools to align them. Nobody had tools to handle them. Nobody had any software to test,” Feldman explained. “And so we have designed this whole manufacturing flow, because nobody has ever done it.” Cerebras’ technology is much more than just the chip it sells — it also includes all of the associated machinery required to actually manufacture and package those chips.

Finally, all that processing power in one chip requires immense power and cooling. Cerebras’ chip uses 15 kilowatts of power to operate — a prodigious amount of power for an individual chip, although relatively comparable to a modern-sized AI cluster. All that power also needs to be cooled, and Cerebras had to design a new way to deliver both for such a large chip.

It essentially approached the problem by turning the chip on its side, in what Feldman called “using the Z-dimension.” The idea was that rather than trying to move power and cooling horizontally across the chip as is traditional, power and cooling are delivered vertically at all points across the chip, ensuring even and consistent access to both.

And so, those were the next three challenges — thermal expansion, packaging and power/cooling — that the company has worked around-the-clock to deliver these past few years.

From theory to reality

Cerebras has a demo chip (I saw one, and yes, it is roughly the size of my head), and it has started to deliver prototypes to customers, according to reports. The big challenge, though, as with all new chips, is scaling production to meet customer demand.

For Cerebras, the situation is a bit unusual. Because it places so much computing power on one wafer, customers don’t necessarily need to buy dozens or hundreds of chips and stitch them together to create a compute cluster. Instead, they may only need a handful of Cerebras chips for their deep-learning needs. The company’s next major phase is to reach scale and ensure a steady delivery of its chips, which it packages as a whole system “appliance” that also includes its proprietary cooling technology.

Expect to hear more details of Cerebras technology in the coming months, particularly as the fight over the future of deep learning processing workflows continues to heat up.

Reputation.com nabs $30M more to help enterprises manage their profiles online

In these days where endorsements from influential personalities online can make or break a product, a startup that’s built a business to help companies harness all the long-tail firepower they can muster to get their name out there in a good way has raised some funding to expand deeper into feedback and other experience territory. Reputation.com, which works with big enterprises in areas like automotive and healthcare to help improve their visibility online and provide more accurate reports to the businesses about how their brands are perceived by customers and others, has raised $30 million in equity financing, money that CEO Joe Fuca said the company will use to continue to expand its tech platform to source more feedback and to future-proof it for further global expansion.

The funding — led by Ascension Ventures, with participation also from new backers Akkadian Ventures, Industry Ventures and River City Ventures and returning investors Kleiner Perkins, August Capital, Bessemer Venture Partners, Heritage Group and Icon Ventures — is the second round Reputation.com has raised since its pivot away from services aimed at individuals. Fuca said the company’s valuation is tripling with this round, and while he wouldn’t go into the details from what I understand from sources (which is supported by data in PitchBook), it had been around $120-130 million in its last round, making it now valued at between $360-390 million now.

Part of the reason that the company’s valuation has tripled is because of its growth. The company doesn’t disclose many customer names (for possibly obvious reasons) but said that three of the top five automotive OEMs and as well as over 10,000 auto dealerships in the U.S. use it, with those numbers now also growing in Europe. Among healthcare providers, it now has 250 customers — including three of the top five — and in the world of property management, more than 100 companies are using Reputation.com. Other verticals that use the company include financial services, hospitality and retail services.

The company competes with other firms that provide services like SEO and other online profile profile management and sees the big challenge as trying to convince businesses that there is more to having a strong profile than just an NPS score (providers of which are also competitors). So, in addition to the metrics that are usually used to compile this figure (based on customer feedback surveys typically), Reputation.com uses unstructured data as well (for example sentiment analysis from social media) and applies algorithms to this to calculate a Reputation Score.

Reputation.com has been around actually since 2006, with its original concept being managing individuals’ online reputations — not exactly in the Klout or PR-management sense, but with a (now very prescient-sounding) intention of providing a way for people to better control their personal information online. Its original name was ReputationDefender and founded by Michael Fertik, it was a pioneer in what came to be called personal information management.

The company proposed an idea of a “vault” for your information, which could still be used and appropriated by so-called data brokers (which help feed the wider ad-tech and marketing tech machines that underpin a large part of the internet economy), but would be done with user consent and compensation.

The idea was hard to scale, however. “I think it was an addressable market issue,” said Fuca, who took over as CEO last year the company was reorienting itself to enterprise services (it sold off the consumer/individual business at the same time to a PE firm), with Fertik taking the role of executive chairman, among other projects. “Individuals seeking reputation defending is only certain market size.”

Not so in the world of enterprise, the area the startup (and I think you can call Reputation.com a startup, given its pivot and restructure and venture backing) has been focusing on exclusively for the better part of a year.

The company today integrates closely with Google — which is not only a major platform for disseminating information in the form of SEO management, but a data source as a repository of user reviews — but despite the fact that Google holds so many cards in the stack, Fuca (who had previously been an exec at DocuSign before coming to Reputation.com) said he doesn’t see it as a potential threat or competitor.

A recent survey from the company about reputation management for the automotive sector underscores just how big of a role Google does play:

Screenshot 2019 08 20 at 11.48.57“We don’t worry about google as competitor,” Fuca said. “It is super attracted to working with partners like us because we drive domain activity, and they love it when people like us explain to customers how to optimise on Google. For Google, it’s almsot like we are an optimization partner and so it helps their entire ecosystem, and so I don’t see them being a competitor or wanting to be.”

Nevertheless, the fact that the bulk of Reputation.com’s data sources are essentially secondary — that is publically available information that is already online and collected by others — will be driving some of the company’s next stage of development. The plan is to start to add in more of its own primary-source data gathering in the form of customer surveys and feedback forms. That will open the door too to more questions of how the company will handle privacy and personal data longer term.

“Ascension Ventures is excited to deepen its partnership with Reputation.com as it enters its next critical stage of growth,” said John Kuelper, Managing Director at Ascension Ventures, in a statement. “We’ve watched Reputation.com’s industry leading reputation management offering grow into an even more expansive CX platform. We’re seeing some of the world’s largest brands and service providers achieve terrific results by partnering with Reputation.com to analyze and take action on customer feedback — wherever it originates — at scale and in real-time. We’re excited to make this additional investment in Reputation.com as it continues to grow and expand its market leadership.”

H2O.ai announces $72.5M Series D led by Goldman Sachs

H2O.ai‘s mission is to democratize AI by providing a set of tools that frees companies from relying on teams of data scientists. Today it got a bushel of money to help. The company announced a $72.5 million Series D round led by Goldman Sachs and Ping An Global Voyager Fund.

Previous investors Wells Fargo, NVIDIA and Nexus Venture Partners also participated. Under the terms of the deal, Jade Mandel from Goldman Sachs will be joining the H2O.ai Board. Today’s investment brings the total raised to $147 million.

It’s worth noting that Goldman Sachs isn’t just an investor. It’s also a customer. Company CEO and co-founder Sri Ambati says the fact that customers, Wells Fargo and Goldman Sachs, have led the last two rounds is a validation for him and his company. “Customers have risen up from the ranks for two consecutive rounds for us. Last time the Series C was led by Wells Fargo where we were their platform of choice. Today’s round was led by Goldman Sachs, which has been a strong customer for us and strong supporters of our technology,” Ambati told TechCrunch.

The company’s main product, H2O Driverless AI, introduced in 2017, gets its name from the fact it provides a way for people who aren’t AI experts to still take advantage of AI without a team of data scientists. “Driverless AI is automatic machine learning, which brings the power of a world class data scientists in the hands of everyone. lt builds models automatically using machine learning algorithms of every kind,” Ambati explained.

They introduced a new recipe concept today, that provides all of the AI ingredients and instructions for building models for different business requirements. H2O.ai’s team of data scientists has created and open sourced 100 recipes for things like credit risk scoring, anomaly detection and property valuation.

The company has been growing since its Series C round in 2017 when it had 70 employees. Today it has 175 and has tripled the number of customers since the prior round, although Ambati didn’t discuss an exact number.  The company has its roots in open source and has 20,000 users of its open source products, according to Ambati.

He didn’t want to discuss valuation and wouldn’t say when the company might go public, saying it’s early days for AI and they are working hard to build a company for the long haul.

Simon Data hauls in $30M Series C to continue building customer data platform

As businesses use an increasing variety of marketing software solutions, the goal around collecting all of that data is to improve customer experience. Simon Data announced a $30 million Series C round today to help.

The round was led by Polaris Partners . Previous investors .406 Ventures and F-Prime Capital also participated. Today’s investment brings the total raised to $59 million, according to the company.

Jason Davis, co-founder and CEO, says his company is trying to pull together a lot of complex data from a variety of sources, while driving actions to improve customer experience. “It’s about taking the data, and then building complex triggers that target the right customer at the right time,” Davis told TechCrunch. He added, “This can be in the context of any sort of customer transaction, or any sort of interaction with the business.”

Companies tend to use a variety of marketing tools, and Simon Data takes on the job of understanding the data and activities going on in each one. Then based on certain actions — such as, say, an abandoned shopping cart — it delivers a consistent message to the customer, regardless of the source of the data that triggered the action.

They see this ability to pull together data as a customer data platform (CDP). In fact, part of its job is to aggregate data and use it as the basis of other activities. In this case, it involves activating actions you define based on what you know about the customer at any given moment in the process.

As the company collects this data, it also sees an opportunity to use machine learning to create more automated and complex types of interactions. “There are a tremendous number of super complex problems we have to solve. Those include core platform or infrastructure, and we also have a tremendous opportunity in front of us on the predictive and data science side as well,” Davis said. He said that is one of the areas where they will put today’s money to work.

The company, which launched in 2014, is based in NYC. The company currently has 87 employees in total, and that number is expected to grow with today’s announcement. Customers include Equinox, Venmo and WeWork. The company’s most recent funding round was a $20 million in July 2018.

Microsoft acquires jClarity, an open source Java performance tuning tool

Microsoft announced this morning that it was acquiring jClarity, an open source tool designed to tune the performance of Java applications. It will be doing that on Azure from now on. In addition, the company has been offering a flavor of Java called AdoptOpenJDK, which they bill as a free alternative to Oracle Java. The companies did not discuss the terms of the deal.

As Microsoft pointed out in a blog post announcing the acquisition, they are seeing increasing use of large-scale Java installations on Azure, both internally with platforms like Minecraft and externally with large customers including Daimler and Adobe.

The company believes that by adding the jClarity team and its toolset, it can help service these Java customers better. “The team, formed by Java champions and data scientists with proven expertise in data driven Java Virtual Machine (JVM) optimizations, will help teams at Microsoft to leverage advancements in the Java platform,” the company wrote in the blog.

Microsoft has actually been part of the AdoptOpenJDK project along with a Who’s Who of other enterprise companies including Amazon, IBM, Pivotal, Red Hat and SAP.

Co-founder and CEO Martin Verburg, writing in a company blog post announcing the deal, unsurprisingly spoke in glowing terms about the company he was about to become a part of. “Microsoft leads the world in backing developers and their communities, and after speaking to their engineering and programme leadership, it was a no brainer to enter formal discussions. With the passion and deep expertise of Microsoft’s people, we’ll be able to support the Java ecosystem better than ever before,” he wrote.

Verburg also took the time to thank the employees, customers and community who has supported the open source project on top of which his company was built. Verburg’s new title at Microsoft will be Principal Engineering Group Manager (Java) at Microsoft.

It is unclear how the community will react to another flavor of Java being absorbed by another large vendor, or how the other big vendors involved in the project will feel about it, but regardless, jClarity is part of Microsoft now.

The Rise of “Bulletproof” Residential Networks

Cybercrooks increasingly are anonymizing their malicious traffic by routing it through residential broadband and wireless data connections. Traditionally, those connections have been mainly hacked computers, mobile phones, or home routers. But this story is about so-called “bulletproof residential VPN services” that appear to be built by purchasing or otherwise acquiring discrete chunks of Internet addresses from some of the world’s largest ISPs and mobile data providers.

In late April 2019, KrebsOnSecurity received a tip from an online retailer who’d seen an unusual number of suspicious transactions originating from a series of Internet addresses assigned to a relatively new Internet provider based in Maryland called Residential Networking Solutions LLC.

Now, this in itself isn’t unusual; virtually every provider has the occasional customers who abuse their access for fraudulent purposes. But upon closer inspection, several factors caused me to look more carefully at this company, also known as “Resnet.”

An examination of the IP address ranges assigned to Resnet shows that it maintains an impressive stable of IP blocks — totaling almost 70,000 IPv4 addresses — many of which had until quite recently been assigned to someone else.

Most interestingly, about ten percent of those IPs — more than 7,000 of them — had until late 2018 been under the control of AT&T Mobility. Additionally, the WHOIS registration records for each of these mobile data blocks suggest Resnet has been somehow reselling data services for major mobile and broadband providers, including AT&T, Verizon, and Comcast Cable.

The WHOIS records for one of several networks associated with Residential Networking Solutions LLC.

Drilling down into the tracts of IPs assigned to Resnet’s core network indicates those 7,000+ mobile IP addresses under Resnet’s control were given the label  “Service Provider Corporation” — mostly those beginning with IPs in the range 198.228.x.x.

An Internet search reveals this IP range is administered by the Wireless Data Service Provider Corporation (WDSPC), a non-profit formed in the 1990s to manage IP address ranges that could be handed out to various licensed mobile carriers in the United States.

Back when the WDSPC was first created, there were quite a few mobile wireless data companies. But today the vast majority of the IP space managed by the WDSPC is leased by AT&T Mobility and Verizon Wireless — which have gradually acquired most of their competing providers over the years.

A call to the WDSPC revealed the nonprofit hadn’t leased any new wireless data IP space in more than 10 years. That is, until the organization received a communication at the beginning of this year that it believed was from AT&T, which recommended Resnet as a customer who could occupy some of the company’s mobile data IP address blocks.

“I’m afraid we got duped,” said the person answering the phone at the WDSPC, while declining to elaborate on the precise nature of the alleged duping or the medium that was used to convey the recommendation.

AT&T declined to discuss its exact relationship with Resnet  — or if indeed it ever had one to begin with. It responded to multiple questions about Resnet with a short statement that said, “We have taken steps to terminate this company’s services and have referred the matter to law enforcement.”

Why exactly AT&T would forward the matter to law enforcement remains unclear. But it’s not unheard of for hosting providers to forge certain documents in their quest for additional IP space, and anyone caught doing so via email, phone or fax could be charged with wire fraud, which is a federal offense that carries punishments of up to $500,000 in fines and as much as 20 years in prison.

WHAT IS RESNET?

The WHOIS registration records for Resnet’s main Web site, resnetworking[.]com, are hidden behind domain privacy protection. However, a cursory Internet search on that domain turned up plenty of references to it on Hackforums[.]net, a sprawling community that hosts a seemingly never-ending supply of up-and-coming hackers seeking affordable and anonymous ways to monetize various online moneymaking schemes.

One user in particular — a Hackforums member who goes by the nickname “Profitvolt” — has spent several years advertising resnetworking[.]com and a number of related sites and services, including “unlimited” AT&T 4G/LTE data services, and the immediate availability of more than 1 million residential IPs that he suggested were “perfect for botting, shoe buying.”

The Hackforums user “Profitvolt” advertising residential proxies.

Profitvolt advertises his mobile and residential data services as ideal for anyone who wishes to run “various bots,” or “advertising campaigns.” Those services are meant to provide anonymity when customers are doing things such as automating ad clicks on platforms like Google Adsense and Facebook; generating new PayPal accounts; sneaker bot activity; credential stuffing attacks; and different types of social media spam.

For readers unfamiliar with this term, “shoe botting” or “sneaker bots” refers to the use of automated bot programs and services that aid in the rapid acquisition of limited-release, highly sought-after designer shoes that can then be resold at a profit on secondary markets. All too often, it seems, the people who profit the most in this scheme are using multiple sets of compromised credentials from consumer accounts at online retailers, and/or stolen payment card data.

To say shoe botting has become a thorn in the side of online retailers and regular consumers alike would be a major understatement: A recent State of The Internet Security Report (PDF) from Akamai (an advertiser on this site) noted that such automated bot activity now accounts for almost half of the Internet bandwidth directed at online retailers. The prevalance of shoe botting also might help explain Footlocker‘s recent $100 million investment in goat.com, the largest secondary shoe resale market on the Web.

In other discussion threads, Profitvolt advertises he can rent out an “unlimited number” of so-called “residential proxies,” a term that describes home or mobile Internet connections that can be used to anonymously relay Internet traffic for a variety of dodgy deals.

From a ne’er-do-well’s perspective, the beauty of routing one’s traffic through residential IPs is that few online businesses will bother to block malicious or suspicious activity emanating from them.

That’s because in general the pool of IP addresses assigned to residential or mobile wireless connections cycles intermittently from one user to the next, meaning that blacklisting one residential IP for abuse or malicious activity may only serve to then block legitimate traffic (and e-commerce) from the next user who gets assigned that same IP.

A BULLETPROOF PLAN?

In one early post on Hackforums, Profitvolt laments the untimely demise of various “bulletproof” hosting providers over the years, from the Russian Business Network and Atrivo/Intercage, to McColo, 3FN and Troyak, among others.

All of these Internet providers had one thing in common: They specialized in cultivating customers who used their networks for nefarious purposes — from operating botnets and spamming to hosting malware. They were known as “bulletproof” because they generally ignored abuse complaints, or else blamed any reported abuse on a reseller of their services.

In that Hackforums post, Profitvolt bemoans that “mediums which we use to distribute [are] locking us out and making life unnecessarily hard.”

“It’s still sketchy, so I am not going all out to reveal my plans, but currently I am starting off with a 32 GB RAM server with a 1 GB unmetered up-link in a Caribbean country,” Profitvolt told forum members, while asking in different Hackforums posts whether there are any other users from the dual-island Caribbean nation of Trinidad and Tobago on the forum.

“To be quite honest, the purpose of this is to test how far we can stretch the leniency before someone starts asking questions, or we start receiving emails,” Profitvolt continued.

Hackforums user Profitvolt says he plans to build his own “bulletproof” hosting network catering to fellow forum users who might want to rent his services for a variety of dodgy activities.

KrebsOnSecurity started asking questions of Resnet after stumbling upon several indications that this company was enabling different types of online abuse in bite-sized monthly packages. The site resnetworking[.]com appears normal enough on the surface, but a review of the customer packages advertised on it suggests the company has courted a very specific type of client.

“No bullshit, just proxies,” reads one (now hidden or removed) area of the site’s shopping cart. Other promotions advertise the use of residential proxies to promote “growth services” on multiple social media platforms including CraigslistFacebook, Google, Instagram, Spotify, Soundcloud and Twitter.

Resnet also peers with or partners with several other interesting organizations, including:

residential-network[.]com, also known as “IAPS Security Services” (formerly intl-alliance[.]com), which advertises the sale of residential VPNs and mobile 4G/IPv6 proxies aimed at helping customers avoid being blocked while automating different types of activity, from mass-creating social media and email accounts to bulk message sending on platforms like WhatsApp and Facebook.

Laksh Cybersecurity and Defense LLC, which maintains Hexproxy[.]com, another residential proxy service that largely courts customers involved in shoe botting.

-Several chunks of IP space from a Russian provider variously known by the names “SERVERSGET” and “Men Danil Valentinovich,” which has been associated with numerous instances of hijacking vast swaths of IP addresses from other organizations quite recently.

Some of Profitvolt’s discussion threads on Hackforums.

WHO IS RESNET?

Resnetworking[.]com lists on its home page the contact phone number 202-643-8533. That number is tied to the registration records for several domains, including resnetworking[.]com, residentialvpn[.]info, and residentialvpn[.]org. All of those domains also have in their historic WHOIS records the name Joshua Powder and Residential Networking Solutions LLC.

Running a reverse WHOIS lookup via Domaintools.com on “Joshua Powder” turns up almost 60 domain names — most of them tied to the email address joshua.powder@gmail.com. Among those are resnetworking[.]info, resvpn[.]com/net/org/info, tobagospeaks[.]com, tthack[.]com and profitvolt[.]com. Recall that “Profitvolt” is the nickname of the Hackforums user advertising resnetworking[.]com.

The email address josh@tthack.com was used to register an account on the scammer-friendly site blackhatworld[.]com under the nickname “BulletProofWebHost.” Here’s a list of domains registered to this email address.

A search on the Joshua Powder and tthack email addresses at Hyas, a startup that specializes in combining data from a number of sources to provide attribution of cybercrime activity, further associates those to mafiacloud@gmail.com and to the phone number 868-360-9983, which is a mobile number assigned by Digicel Trinidad and Tobago Ltd. A full list of domains tied to that 868- number is here.

Hyas’s service also pointed to this post on the Facebook page of the Prince George’s County Economic Development Corporation in Maryland, which appears to include a 2017 photo of Mr. Powder posing with county officials.

‘A GLORIFIED SOLUTIONS PROVIDER’

Roughly three weeks ago, KrebsOnSecurity called the 202 number listed at the top of resnetworking[.]com. To my surprise, a man speaking in a lovely Caribbean-sounding accent answered the call and identified himself as Josh Powder. When I casually asked from where he’d acquired that accent, Powder said he was a native of New Jersey but allowed that he has family members who now live in Trinidad and Tobago.

Powder said Residential Networking Solutions LLC is “a normal co-location Internet provider” that has been in operation for about three years and employs some 65 people.

“You’re not the first person to call us about residential VPNs,” Powder said. “In the past, we did have clients that did host VPNs, but it’s something that’s been discontinued since 2017. All we are is a glorified solutions provider, and we broker and lease Internet lines from different companies.”

When asked about the various “botting” packages for sale on Resnetworking[.]com, Powder replied that the site hadn’t been updated in a while and that these were inactive offers that resulted from a now-discarded business model.

“When we started back in 2016, we were really inexperienced, and hired some SEO [search engine optimization] firms to do marketing,” he explained. “Eventually we realized that this was creating a shitstorm, because it started to make us look a specific way to certain people. So we had to really go through a process of remodeling. That process isn’t complete, and the entire web site is going to retire in about a week’s time.”

Powder maintains that his company does have a contract with AT&T to resell LTE and 4G data services, and that he has a similar arrangement with Sprint. He also suggested that one of the aforementioned companies which partnered with Resnet — IAPS Security Services — was responsible for much of the dodgy activity that previously brought his company abuse complaints and strange phone calls about VPN services.

“That guy reached out to us and he leased service from us and nearly got us into a lot of trouble,” Powder said. “He was doing a lot of illegal stuff, and I think there is an ongoing matter with him legally. That’s what has caused us to be more vigilant and really look at what we do and change it. It attracted too much nonsense.”

Interestingly, when one visits IAPS Security Services’ old domain — intl-alliance[.]com — it now forwards to resvpn[.]com, which is one of the domains registered to Joshua Powder.

Shortly after our conversation, the monthly packages I asked Powder about that were for sale on resnetworking[.]com disappeared from the site, or were hidden behind a login. Also, Resnet’s IPv6 prefixes (a la IAPS Security Services) were removed from the company’s list of addresses. At the same time, a large number of Profitvolt’s posts prior to 2018 were deleted from Hackforums.

EPILOGUE

It appears that the future of low-level abuse targeting some of the most popular Internet destinations is tied to the increasing willingness of the world’s biggest ISPs to resell discrete chunks of their address space to whomever is able to pay for them.

Earlier this week, I had a Skype conversation with an individual who responded to my requests for more information from residential-network[.]com, and this person told me that plenty of mobile and land-line ISPs are more than happy to sell huge amounts of IP addresses to just about anybody.

“Mobile providers also sell mass services,” the person who responded to my Skype request offered. “Rogers in Canada just opened a new package for unlimited 4G data lines and we’re currently in negotiations with them for that service as well. The UK also has 4G providers that have unlimited data lines as well.”

The person responding to my Skype messages said they bought most of their proxies from a reseller at customproxysolutions[.]com, which advertises “the world’s largest network of 4G LTE modems in the United States.”

He added that “Rogers in Canada has a special offer that if you buy more than 50 lines you get a reduced price lower than the $75 Canadian Dollar price tag that they would charge for fewer than 50 lines. So most mobile ISPs want to sell mass lines instead of single lines.”

It remains unclear how much of the Internet address space claimed by these various residential proxy and VPN networks has been acquired legally or through other means. But it seems that Resnet and its business associates are in fact on the cutting edge of what it means to be a bulletproof Internet provider today.

The Good, the Bad and the Ugly in Cybersecurity – Week 33

The Good

U.S. Cyber Command continues to fight the good fight by sharing malware samples with the rest of the industry. This week, the military unit shared samples of ElectricFish malware, a tunneling tool discovered in May and attributed to APT38, a North Korean affiliated group said to focus on financial theft. ElectricFish is a Windows 32-bit command line tool that enables attackers to use tunneling to exfiltrate data from a backdoored system.

image of electric fish malwareImage Source

Nice to see some good news this week concerning Bluetooth, particularly after the recent revelation that iOS devices allow snoops to leverage the protocol to surreptitiously gather device data. Turning the tables, researchers have developed an app that can sniff out credit card skimmers at gas stations. Dubbed ‘Bluetana‘, the app is currently only available to gas pump inspectors but has already been used to flush out 64 skimmers in Arizona, California, Nevada and Maryland. 

image of bluetana app

Image Source

The Bad

Long before most people started to think too much about computer security – well, 2001 anyway – Microsoft introduced a nice feature for multilingual users of Office XP. The feature helpfully pops up a small ‘composition’ window that tries to predict a foreign language character (think Simplified Chinese, Korean or Japanese characters) based on the Roman characters you type. Sounds great. Fast forward to 2019 and Project Zero’s discovery that the underlying protocol for this feature – the Text Services Framework – exposes users of every version of Windows ever since, regardless of whether you have installed foreign language support or not, to a privilege elevation vulnerability. An attacker needs login credentials first (so, no RCE here), but any user can quickly and reliably elevate to root. The bug is fixed in the latest available patch and all Windows users are urged to patch without delay. 

image of tavis ormandy tweetImage Source

The Ugly

Kaspersky Labs – who’ve been out-of-favor with the US Government since late 2017 – have been exposing their users to the possibility of cross-site internet tracking. Kaspersky products intended to provide internet security – Kaspersky Antivirus, Internet Security, Total Security, Free Antivirus, and Small Office Security – were found to be injecting a uniquely identifying string into every web page a user visits. The string is part of a URL that locates a remotely-hosted JavaScript that Kaspersky use to determine if a webpage is malicious or not. As each user was assigned a unique, persistent string, websites and malicious scripts could track the user from one site to another. Kaspersky patched the flaw (CVE-2019-8286) several weeks prior to the bug being disclosed by replacing the UUID-style string with a fixed, generic string for all users.

image of kaspersky flawImage Source

Huawei, another company that are not in the US government’s good books at the moment, have reportedly been aiding at least two African nations in their attempts to suppress political opponents and restrict internet freedoms. The company’s employees apparently lent their assistance to intelligence agencies in Zambia and Uganda to hack WhatsApp and Skype communications and shutdown news sites unfavorable to the Ugandan government. For their part, Huawei have denied the allegations.


Like this article? Follow us on LinkedIn, Twitter, YouTube or Facebook to see the content we post.

Read more about Cyber Security

Alibaba cloud biz is on a run rate over $4B

Alibaba announced its earnings today, and the Chinese e-commerce giant got a nice lift from its cloud business, which grew 66% to more than $1.1 billion, or a run rate surpassing $4 billion.

It’s not exactly on par with Amazon, which reported cloud revenue of $8.381 billion last quarter, more than double Alibaba’s yearly run rate, but it’s been a steady rise for the company, which really began taking the cloud seriously as a side business in 2015.

At that time, Alibaba Cloud’s president Simon Hu boasted to Reuters that his company would overtake Amazon in four years. It is not even close to doing that, but it has done well to get to more than a billion a quarter in just four years.

In fact, in its most recent data for the Asia-Pacific region, Synergy Research, a firm that closely tracks the public cloud market, found that Amazon was still number one overall in the region. Alibaba was first in China, but fourth in the region outside of China, with the market’s Big 3 — Amazon, Microsoft and Google — coming in ahead of it. These numbers were based on Q1 data before today’s numbers were known, but they provide a sense of where the market is in the region.

Screenshot 2019 08 15 11.17.26

Synergy’s John Dinsdale says the company’s growth has been impressive, outpacing the market growth rate overall. “Alibaba’s share of the worldwide cloud infrastructure services market was 5% in Q2 — up by almost a percentage point from Q2 of last year, which is a big deal in terms of absolute growth, especially in a market that is growing so rapidly,” Dinsdale told TechCrunch.

He added, “The great majority of its revenue does indeed come from China (and Hong Kong), but it is also making inroads in a range of other APAC country markets — Indonesia, Malaysia, Singapore, India, Australia, Japan and South Korea. While numbers are relatively small, it has also got a foothold in EMEA and some operations in the U.S.”

The company was busy last quarter adding more than 300 new products and features in the period ending June 30th (and reported today). That included changes and updates to core cloud offerings, security, data intelligence and AI applications, according to the company.

While the cloud business still isn’t a serious threat to the industry’s Big Three, especially outside its core Asia-Pacific market, it’s still growing steadily and accounted for almost 7% of Alibaba’s total of $16.74 billion in revenue for the quarter — and that’s not bad at all.

Gootkit Banking Trojan | Deep Dive into Anti-Analysis Features

The Gootkit Banking Trojan was discovered back in 2014, and utilizes the Node.JS library to perform a range of malicious tasks, from website injections and password grabbing, all the way up to video recording and remote VNC capabilities. Since its discovery in 2014, the actors behind Gootkit have continued to update the codebase to slow down analysis and thwart automated sandboxes. This post will take a look into the first stage of Gootkit, which contains the unpacking phase and a malicious downloader that sets up the infected system, and its multiple anti-analysis mechanisms.

image of gootkit

Unpacking


MD5 of Packed Sample: 0b50ae28e1c6945d23f59dd2e17b5632

With this specific sample, the unpacking routine is fairly trivial, as it performs self-injection. Simply put, the unpacker will:

Allocate a region of memory -> Decrypt shellcode and copy to the allocated region -> Execute the shellcode, decrypting the first stage Gootkit executable -> Overwrite unpacked with decrypted executable -> Change protections on the decrypted executable and transfer execution to it.

Therefore, in order to unpack it, place breakpoints on both VirtualAlloc and VirtualProtect, and look out for executable headers appearing in the allocated regions of memory.

image of gootkit module kernel32.dll

MD5 of Unpacked Sample: c342af62302936720e52679bc431d5a8

Immediately upon opening the sample in IDA, you’ll notice the use of the CreateThread API – this is used excessively throughout the binary, potentially as an anti-dynamic analysis method. It becomes quite difficult to debug the program due to the fact that multiple threads are running at once; however, this can be avoided by focusing on one thread per execution. Static analysis methods are also hindered, due to the levels of obfuscation utilized by the sample. Whilst there are quite a few strings in plaintext, nearly all of the important strings used are decrypted at run time, using a simple but effective XOR algorithm. Not only are the strings encrypted, they are also stored as stack strings, making it more complex to extract the important data. 

As mentioned previously, the algorithm is fairly simple. Essentially what happens is there are 2 different “strings”. The first string (typically shorter), will loop around, XOR’ing each byte with a byte of the second string. An example of this algorithm in Python can be seen below.

image of gootkit string decryption

The example above will return the string kernel32.dll.

Before Gootkit begins to perform its malicious routines, it first checks the arguments passed to it – this determines the path it follows. The possible arguments that Gootkit accepts are:

--reinstall
--service
-test
--vwxyz

If no argument is given, Gootkit will perform a setup routine, and then execute itself with the --vwxyz argument. The -test argument simply causes the process to exit, whereas the --reinstall argument will reinstall Gootkit using the persistence method that we will be covering in the next post. Finally, the --service argument will simply set an additional environment variable, specifically the variable name USERNAME_REQUIRED, with the value set as TRUE. In this post we, will be focusing primarily on the setup phase, to understand the steps Gootkit takes before executing itself with the --vwxyz argument.

Anti-Analysis Functionality

As mentioned previously, Gootkit packs plenty of Anti-Analysis features to evade sandboxes, prevent execution in a Virtual Machine, and slow down analysis. Interestingly, the functions responsible for these features are skipped if a specific environment variable is set. The variable that is set during runtime is named crackmeololo, and the value given to it is navigator. When it comes to checking the value, rather than compare it to a string, Gootkit will utilize CRC-32/JAMCRC hashing in order to check the validity. If the CRC hashes don’t match, the system checks begin.

image of gootkit crackmeololo

The first check that Gootkit performs is a filename check. Simply put, there is a hardcoded list of CRC hashed filenames inside the binary, which are compared against the hash of the current filename. If a match is found, Gootkit will create a batch file that will delete the original executable. The process will then exit. A list of the filenames that Gootkit searches for can be seen below.

SAMPLE.EXE
MALWARE.EXE
BOT.EXE
SANDBOX.EXE
TEST.EXE
KLAVME.EXE
MYAPP.EXE
TESTAPP.EXE

image of gootkit filename check

The next checks are performed almost immediately after the filename check. Gootkit will create another thread, where it will output the string “MP3 file corrupted” using OutputDebugStringA, and then check the environment variable crackmeololo once again. If the CRC hashes match, it will continue on to decrypt the on board configuration – if not, it will perform a more in depth check of the environment.

First, it begins by opening the registry key HardwareDESCRIPTIONSystemCentralProcessor, and then queries the ProcessorNameString, comparing the value to Xeon. The Xeon processor is used in servers primarily, and not in laptops or desktops. This is a good indicator that the malware is running in a sandbox, so if it is detected, Gootkit will enter an endless sleep-loop cycle.

image of gootkit infinite sleep

If Xeon is not detected, execution will resume; however, the next check is a lot more intensive. Similar to the filename check, Gootkit also contains a hardcoded list of MAC address identifiers used to detect sandboxes or VMs. After loading RPCRT4.DLL, it will call UuidCreateSequential, which uses the MAC Address to create a GUID. If any of the values match, it will enter an infinite sleep-loop cycle once again. A list of the hardcoded MAC Addresses along with the corresponding vendors can be seen below.

F01FAF00 Dell
00505600 VMWare
8002700 PCS System Technology GmbH
000C2900 VMWare
00056900 VMWare
0003FF00 Microsoft
001C4200 Parallels
00163E00 XenSource

image of gootkit check MAC address

Next, Gootkit will call GetModuleHandleA in an attempt to get a handle to either dbghelp.dll and sbiedll.dll, in an attempt to detect a present debugger or the sandbox Sandboxie. If a handle is returned successfully, an infinite sleep cycle will occur. Continuing on, the current username will be retrieved with a call to GetUserNameA, and compared to CurrentUser and Sandbox. The computer name will then be retrieved and compared to SANDBOX and 7SILVIA. As you may have guessed, if any of these match, the sample will enter into an infinite sleep cycle.

image of gootkit detect sandbox

Continuing on, Gootkit will query HARDWAREDESCRIPTIONSystemSystemBiosVersion and compare the value to; AMI, BOCHS, VBOX, QEMU, SMCI,  INTEL  – 6040000, FTNT-1, and SONI. Once again, match = infinite sleep cycle. 

image of gootkit check bios version

Yet another registry query is performed, this time with the key HARDWAREDescriptionSystemVideoBiosVersion, with the value being compared to VirtualBox. Finally, it queries  SOFTWAREMicrosoftWindowsCurrentVersionSystemBiosVersion or HARDWAREDESCRIPTIONSystemSystemBiosVersion for 3 values that correspond to Joe Sandbox and CWSandbox:

55274-640-2673064-23950: Joe Sandbox
76487-644-3177037-23510: CWSandbox
76487-337-8429955-22614: CWSandbox

If all checks are passed, then execution of the sample will continue, by setting up persistence and retrieving the payload from the C2 server. Before doing that, it will check its filename once again, using the same CRC hashing we saw earlier.

In the next post, we will take a look at the persistence method used by Gootkit, and take a look at the --reinstall pathway, as well as the communications routine used by the sample to retrieve the final stage.


Like this article? Follow us on LinkedIn, Twitter, YouTube or Facebook to see the content we post.

Read more about Cyber Security

Microsoft Azure CTO Mark Russinovich will join us for TC Sessions: Enterprise on September 5

Being the CTO for one of the three major hypercloud providers may seem like enough of a job for most people, but Mark Russinovich, the CTO of Microsoft Azure, has a few other talents in his back pocket. Russinovich, who will join us for a fireside chat at our TechCrunch Sessions: Enterprise event in San Francisco on September 5 (p.s. early-bird sale ends Friday), is also an accomplished novelist who has published four novels, all of which center around tech and cybersecurity.

At our event, though, we won’t focus on his literary accomplishments (except for maybe his books about Windows Server) as much as on the trends he’s seeing in enterprise cloud adoption. Microsoft, maybe more so than its competitors, always made enterprise customers and their needs the focus of its cloud initiatives from the outset. Today, as the majority of enterprises is looking to move at least some of their legacy workloads into the cloud, they are often stumped by the sheer complexity of that undertaking.

In our fireside chat, we’ll talk about what Microsoft is doing to reduce this complexity and how enterprises can maximize their current investments into the cloud, both for running new cloud-native applications and for bringing legacy applications into the future. We’ll also talk about new technologies that can make the move to the cloud more attractive to enterprises, including the current buzz around edge computing, IoT, AI and more.

Before joining Microsoft, Russinovich, who has a Ph.D. in computer engineering from Carnegie Mellon, was the co-founder and chief architect of Winternals Software, which Microsoft acquired in 2006. During his time at Winternals, Russinovich discovered the infamous Sony rootkit. Over his 13 years at Microsoft, he moved from Technical Fellow up to the CTO position for Azure, which continues to grow at a rapid clip as it looks to challenge AWS’s leadership in total cloud revenue.

Tomorrow, Friday, August 16 is your last day to save $100 on tickets before prices go up. Book your early-bird tickets now and keep that Benjamin in your pocket.

If you’re an early-stage startup, we only have three demo table packages left! Each demo package comes with four tickets and a great location for your company to get in front of attendees. Book your demo package today before we sell out!