Outages
The year started with problems. Slack suffered a four-hour downtime when people returned to work from Christmas break. Even local Finnish newspaper reported downtime for 10 million users and 750,000 companies, reflecting the importance of the service to business operations.
The media joked that Slack had forgotten to put the cloud auto-scaling back on after reducing capacity for the Christmas break. That was a partial truth. Slack did not publish a public post-mortem yet, but the problem was reported to be the slowness on AWS’s Transit Gateway scaling. The service did not keep pace with the growth in demand and the network problem began to accumulate in the servers and the monitoring and provisioning system. The problem was solved by fixing the provisioning system. Slack’s own explanation was published later.
In this case, it was once again seen how network problems have a wide-ranging impact on different systems and parts of the environment. When control and management are lost, it is not known what the situation is and the problem is difficult to correct. In the case of Slack, demand could have been anticipated earlier and the service scaled more evenly. The application’s logic led to a DDoS attack when users bombarded servers with connection requests. Of course, the problems are fixed and this can never happen again. The problem is that Slack and the other companies are not prepared for the worst. What feels like impossible or incredible progress can happen during cascading failures. So don’t underestimate the impossible possibilities and prepare for the worst-case scenario for real.
Cyber security
In the case of Solarwinds, the laundry will continue. In addition to security companies Malwarebytes, Microsoft, FireEye, CrowdStrike, a DNS-based list of Phase 2 attacks on 23 corporate domains was released. Only a few companies have been of interest.
The stolen data was allegedly on sale on the solarleaks page for a total price of $ 1M. The price of individual company data ranged from $ 50,000 to $ 600,000. The prices were considered so amusing that buyers could hardly be found and the whole data leak was considered a bit questionable. It was more about misleading and creating chaos, which has been seen from Russian actors.
Solarwinds has long been known to have shortcomings and bad practices that contributed to the attack. And Solarwinds is not alone with that. In general, private investors behind companies, such as Orlando Bravo, make millions by exploiting companies. Investors are researching all possible savings targets and ripping them out of the “unnecessary” ones. The financier peels off the cream, and the risks and survival are left to the companies. It is no wonder that there are shortcomings in product quality and company practices.
From the case, we can learn that the advanced attack comes indirectly and through many phases. Third parties and partners are often involved in the chain. Therefore, internal resources must be protected also. Workflows and practices need to be reviewed and controls improved. The application development build process is a critical production system and should be treated that way. The process must always produce the same verified results. The transition of the industry from black box software to more transparent and secure code needs to be strengthened. Almost all software contains open source code, but the problem is that most contain outdated code. The most dangerous code is flooded in with the modified library. Users should be able to ask software bill of materials that lists the components and versions used.
Vulnerabilities have again been found in Zyxel and Sonicwall firewalls, Cisco SD-WAN, Ubiquiti cloud service, Linux DNS forward client and sudo. Backdoor vulnerabilities are still emerging. If products are backdoor ported and security is compromised, the vulnerabilities never end.
Windows RDP is used for DDoS attacks. There are over 33,000 public RDP servers in the world to take advantage of. The gain factor of the RDP is 86, which settles at the top end of the efficiency, although not close to the Memcached factor.
In the fight against cyber criminals, Europol and a consortium of different countries have taken over Emotet, the world’s largest and most dangerous botnet. Hopefully this will decrease crime or at least makes it harder for criminals. But probably just for a short time.
In health care, threats are being fought with great stakes. There is some sort of ethic among the attackers that human lives are not threatened, even if there are easy ways to do so. The motivator is still money and the threats are more related to information leaks and blackmailing. The other story may be a time of crisis or if a state actor decides to do harm. In any case, there is a real risk of cyber disruption. In the future, attacks will also target medical research and healthcare partners. As a result, the effects can be unpredictable and widespread.
Palo Alto strikes healthcare IoT protection with its own product. The problems with security products in healthcare are similar to those in OT environments for industrial automation: network availability and continuous operation must be ensured and interruptions are not allowed. Therefore, bringing active security into the hospital environment is difficult. Also resourcing of IT staff and equipment is often weaker in the healthcare sector than in other industries.
Internet
BGP routing on the Internet regularly leaks. On January 6, 2021, nearly 9,000 routes leaked from Hong Kong through the British operator. This time, the disruption had little effect on traffic as only 1/3 of the operators accepted the false advertising. Only 1% of the wrong routes reached global routing. I don’t know if route RPKI validation played a part in that. In any case, RPKI has gained visibility recently with the deployment of major ISPs. RPKI ensures the correct origin of the routes and can partly prevent false advertisements and hijackings. Of course, there are many other practices involved in the functionality and security of Internet routing, and RPKI alone will not save in every case.
About 30% of the prefixes are RPKI-signed. Europe and the Middle East are clearly leading the deployment, and Asia is following suit. Cloud services have also brought their cards into the pile. AWS, as the world’s largest owner of IP addresses, adopted RPKI. The magnitude is indicated by the fact that AWS has about 100 million IPv4 addresses. According to statistics, the market price of a single address could be $ 25, so the value of IPv4 addresses would be a staggering $ 2.5 billion. Half of the addresses are in use and you can find the list here.
In Internet routing, the number of BGP updates is growing steadily. The exact cause is not known, but it is probably related to network fragmentation and traffic diversion. Routers have to handle a lot of routing changes, which requires more performance from them. The Internet is starting to become more unstable all the time as routing changes are constantly happening.
There are many problems with routing optimization. A special false advertisement containing 16 AS-PATH prepend spread because it was more specific (/24) than the original route (/23). The long AS path didn’t make route less preferable. Instead, it just looked crazy, whatever might the purpose have been.
The BGP protocol has grown for many different use cases, but it still doesn’t satisfy agile operators. The initiative now is to make BGP a fully extensible and programmable protocol xBGP, so that users can develop the features they need, regardless of slow standardization. The plugin model would open a closed protocol for additional plugins.
NaaS
SASE is much on display and there seems to be a demand. The idea makes sense, cost and ease are SASE’s selling points. But in terms of implementation, the issue is much more complicated. SASE’s idea is that business traffic would focus mainly on cloud and SaaS services, making it sensible to provide security services over the Internet. However, the operations of the companies are quite a lot of the hybrid model, where some of the operations run on their own premises and local security is needed there. Transporting traffic back and forth between the cloud and your own premises is not optimal or even possible. Also, apps haven’t suddenly turned completely cloud-native, even if they run in the cloud. And the situation is not going to change any time soon.
SASE is not a substitute for all security and one manufacturer cannot provide everything for a wide range of business needs, even if a SASE supplier wants to lock the customer into its own ecosystem. Products are new and may be unfinished. SD-WAN is still one part of the solution and many manufacturers have joined forces with different SD-WAN manufacturers. SD-WAN is also available either on devices or from the cloud. Looking to the future, the situation seems unsustainable. More and more security and content players are building their own global service network. This will sooner or later lead to mergers and acquisitions.
For SD-WAN, there is also an open source option for the DIY builders with FlexiWAN SD-WAN and pfSense firewall software. The solution, which is mainly intended for MSP use by operators and integrators, is now packaged in a virtualized CPE kit.
The Packet Fabric Cloud Router connects multi-clouds and data centers. The product is part of a set of multi-cloud solutions that attempt to manage a complex distributed environment. Other manufacturers include e.g. Alkira, Aviatrix, Arrcus, Drivenets, Pureport, Volta Networks.
F5 acquires Volterra, which specializes in distributed cloud and Edge-as-a-Service platform, which competes with, for example, CDNs for secure application distribution. Volterra is backed by the developers of Contrail. F5 has long since moved from traditional iron load balancing to cloud application development. Edge is not built on hardware, but on software.
Edge is a place no one knows. It is where the operator and the cloud are, but the concrete is very unclear. Forrester predicts a turning point for Edge this year. The edge would suddenly appear and begin to gnaw at the edge of the cloud. In fact, the edge is a cloud that is but scattered close to the user. Growth is slow so far and no rapid change is in sight.
In the Telco cloud, Israeli Drive Nets has risen to be a hot manufacturer and unicorn status with a billion valuation. The company makes cloud-based software for network routing and switching. The customers are 36 large operators and content providers, the largest being AT&T. Again, the new challenger attacks traditional hardware manufacturers and vows to cloud-based software and virtualization. Maybe there’s still a little way to go, but the interest is high and Network-as-a-Service (NaaS) is the direction of the future. SD-WAN and SASE will be the same trend.
The use of open source in the operator network is increasing. The BNG software used for customer termination is now available as open source. DT has implemented OpenBNG software.
5G
In the operator field, public clouds are chasing telcos to gain a foothold in the 5G RAN. Telcos have handed over their own cloud structure and are looking for a solution from a public cloud. But even the three cloud giants do not have a decentralized, real-time, and consistent platform suitable for operator use that would meet tough capacity, latency, and availability requirements. Nokia signed a partnership agreement with Google to develop a cloud-based 5G network.
Open RAN technology is gaining support as operators are now looking for a replacement for Huawei, which is banned in many places, both in terms of technology and cost. The Open RAN is now also pursuing a political agenda against Huawei, but Nokia and Ericsson are also on the line. In Australia, Huawei has already looked to the future and is now investing in 6G.
In 5G, new use cases are waiting for the time being and sales are mainly more data for subscribers. There are high expectations for new business applications, although companies are lazy to innovate. China has introduced an industrial program to connect industry to 5G. It is already partly ready, although there is not much information about the experiences. It is good for companies to think about what is the reality, for example, in terms of terminal and connection costs, business models, network coverage and reliability. At the same time, tens of millions of pounds are being spent in the UK to promote 5G technology to the public.
European countries are lagging in the security practices that the EU has adopted for 5G networks. The pace of implementation of the toolkit is very different. 2/3 of the countries have identified high-risk actors, but only 1/3 have done something about it, even though ready-made instructions have been offered. The British NCSC also publishes network security guidelines for telcos. The most important issue is the supply chain. In this context, it is generally good to identify supply chain risks: dependencies, product security, and access control to systems and data.
At the same time as Google and Facebook are abandoning their own broadband projects, the EU is painting its own satellite broadband plans to provide decent connections to sparsely populated areas. The problem with these projects has been that the “next billion” users from developing countries, or tens of millions users from rural Europe, are paying far too much. The old truth in everything is that the last 10-20% is so expensive and laborious to achieve that it is not worth pursuing.
In IoT networks, LoRaWAN develops and makes roaming possible between 27 countries around the world.
Operation
Cisco unveiled its new design box, the microswitch. It has evoked mixed feelings but the cute outlook clearly appeals. The idea is to bring the fiber to your desk or near equipment and connect this 4-port switch to the end of the fiber. The switch does have nice features, e.g. full intent-based Networking DNA Center nation, POE +, USB-C power supply. I can’t escape the idea that a nice product has now been invented to spread millions of units around campuses and thereby promote the use of the DNA Center and SDA. There are certainly a lot of real use cases as well.
Where did intent-based networking go? The hype went and the thing became mundane. IBN has become part of management systems. The abstraction of network components, the definition of relationships, the collection of telemetry data are prerequisites for the operation of a self-directed network. IBN has become a major visibility tool that is really in demand in complex networks.
Today’s networks still have an insane amount of useless design and configuration that has its roots in adjusting network properties and protocols. The future may look different. Traditional building blocks and functions can change completely. Why, in general, would we want to adjust and control everything manually when we are obviously no longer able to do so? For example, network monitoring and automation is often insanely cumbersome coding and constant updating and tuning. Procedures need to change. In this sense, NaaS is the future. On the other hand, the responsibility also lies with the manufacturers, as not all uses and objects are the same, no matter how desired. Manufacturers must allow a certain amount of customizability to meet the user’s needs. However, the user must also make an attitude change and trust a certain number of functions to be handled automatically by the system. Let’s take advantage of automation and move on to the next level because that’s where we are needed.
Trade policy
Cisco’s purchase of Acacia, which had been protracted for almost a couple of years, took an exciting turn when the Chinese government had not approved the deal for fear of the future of Acacia’s large Chinese customers. Legal challenges were swayed, the deal was canceled, the price was raised and the agreement was made a week later. Cisco paid 4.5 billion instead of the original 2.6 billion and the Chinese were happy. Coherent optics is very important to Cisco. It gets Cisco into the operator game from which it has been losing. Before the approval of the transaction, Telia Carrier had time to announce that it would introduce Cisco-Acacia 400G pluggables in routers and run the DWDM line through the open Open Line System.
The idea is to transfer the DWDM transmitter directly to the SFP module of the router and drive the wavelength through an open multivendor mux to the fiber. The biggest benefits of the solution are flexibility and cost and energy savings. In routers, Telia has moved away from the vendor proprietary chips to Broadcom’s DNX chipset, which already produces 70% of capacity. Traditional Cisco and Juniper routers with their own chips have quickly fallen into the minority.
Co-packaged optics and photonics are also hot topics among circuit manufacturers. Power consumption and heat production are the biggest problems in increasing capacity today. Therefore, the technique is tried to be squeezed as close together as possible. This, on the other hand, means that the flexibility of the interchangeable optics is lost. Cisco teamed up directly with Inphi and Broadcom unveiled its own solution.
The Acacia case shows how uncertain you can be with technology choices, partners, and the supply chain. Trade policy affects the operations of many manufacturers and operators and should be recognized and taken into account in the planning of operations and in making choices and purchases. According to the interviews, 83% of companies said they were now more aware of supply chain risks and thought of ways to manage them. Better information and transparency throughout the chain would improve operational performance and reduce business risk. This would require digitalization and collaboration with manufacturers, wholesalers and retailers.
China’s influence is also reflected in the operations of Ericsson and Nokia. China is the fastest growing market and the ban on Huawei in Sweden raised a fuss as Ericsson feared repercussions on its sales in China. CEO Ekholm said Sweden is a bad country for them and put strong pressure to cancel the Huawei ban. Discrimination against Huawei can very likely lead to discrimination against Western countries in China and even the division of industry and standards.
It happened before. The story of Cisco’s Linksys purchase in the early 2000s tells how the Linksys router had an open source Linux operating system, the source code of which, however, was not released under the license. Cisco and Linksys did not know about this and apparently Linux had put into the device through the subcontractor chain of circuit maker Broadcom and the information didn’t reach Linksys and Cisco. The issue then came public and Cisco took care of the license issue. Linux survived and became a favorite tool for hackers and the WRT54G router gained a reputation for the legend.
Open source
Elastic perceived the Elastisearch service provided by AWS as an exploitation of the free license for its own software and decided to change the license terms for its software. The intention was clearly to penalize AWS, which was considered not sufficiently contributed back to the development. AWS then decided to branch out from Elastisearch into its own project, which it will maintain and develop itself. At Elastic, the thing went personal and it did a disservice to the community by shutting down its open source project to some degree. AWS took advantage of its size and exploited the free code for its own commercial purposes, perhaps beyond the bounds of good taste. Both parties argue that the changes will not cause many inconveniences to users, but this is not the optimal solution for users of the service either.
Redhat’s CentOS was dug up and RHEL was handed out for free to less than 16 server environments, which can be considered as almost a joke. CentOS is the third most popular distribution on the Internet, covering 17% of servers. That’s ten times what RHEL does. Redhat explained the end of CentOS support by allocating resources to other projects. Another problem was the invisibility of CentOS users. The intention was not to transfer customers to a paid service, but that is exactly what happened. CentOS Stream is no longer the stable reliable version that CentOS was.
There has been movement elsewhere as well. Grafana Cloud announced a free plan and Gitlab removed the starter plan, raising the price fivefold. There is a risk in these if you commit to a certain platform and start building production on it. At least it’s good to be prepared that the price and features can suddenly change. Nokia’s Jonne Soininen gave a good introduction to the use of open source on networking.
The popular Netbox IPAM and DCIM program was backed by corporate support when developer Jeremy Stretch began offering commercial support and development for his program.
Companies
The departure of Vmware CEO Pat Gelsinger for Intel has attracted a lot of attention. Gelsinger is known as one of the best leaders as both a technological visionary and an operational performer. In 8 years, he had time to modify the company considerably, triple the result, make 30 acquisitions and take the company to the network, cloud, security, 5G and containers. Among Gelsinger, several executives have left Vmware. Besides, there has long been a join-or-split tug of war between Vmware and Dell. Now it remains to be seen in which direction the company will proceed. The new leader is left with a pile of miscellaneous technology that should be combined into a useful entity somehow. One man can have a surprisingly big impact on the future of a company and its products.
Epic Failure
Extreme missed Flash’s death date on January 12, 2021, and its network management product WiNG Manager ceased operations in the browser. As a solution, Extreme suggested moving the computer clock backward. Of course, traditional CLI worked as expected.
loopback1.net 