Operating system upgrades at LinkedIn’s scale
Introduction
Completing recurring operating system (OS) upgrades on time and without impacting users can be challenging. For LinkedIn, completing these upgrades at a massive scale has its own complexities as we’re often facing multiple upgrades. To secure our platform and protect our members’ data, we needed a fast and reliable OS upgrade framework with little to no human intervention.
In this blog, we’ll introduce a newly developed system, Operating System Upgrade Automation (OSUA), which allows LinkedIn to scale OS upgrades. OSUA has been used for more than 200,000 upgrades on servers that host LinkedIn’s applications.
Key features
After learning the lessons from the past upgrades, here are four remarkable features that OSUA provides.
Zero impact
One of our key values at LinkedIn is putting our customers and members first. In engineering, this means site-up (linkedin.com can be accessed and served anytime from anywhere securely) is always our first priority. OSUA is designed with mechanisms to ensure that no user-facing impact is risked during OS upgrades on servers. Therefore, zero-impact always comes first before any other features in design decisions.
High throughput
LinkedIn has a growing footprint in its on-prem serving facilities that consists of hundreds of thousands of physical servers. To perform a timely upgrade, OSUA has a high upgrade throughput by leveraging parallelization and some applications’ ephemerality that does not sacrifice site-up or cause any performance regression in the middle of upgrades. During the most recent fleet level upgrades, OSUA was capable of upgrading more than 10x more hosts per day than the old mechanisms before. Currently, we are working towards the goal of upgrading 2x of what OSUA can do now.
Support heterogeneous environments
The LinkedIn serving environment consists of various natures that include, but are not limited to, stateless applications, stateful systems (explained in detail later in the post), infrastructure services, etc. They are hosted in multiple locations, managed by a variety of schedulers, ranging from Rain and Kubernetes to Yarn, and most are deployed in a multi-tenant fashion. OSUA currently supports approximately 94% of the LinkedIn footprint that is made up by these systems and its coverage continues to increase.
Automation, autonomous, reduce toil
In the past years, LinkedIn has undergone a few company-wide server OS upgrades for purposes like tech refreshes or improving our platform’s security. Our previous processes for OS upgrades were highly human resource intensive, which added a significant amount of toils to the teams involved. To overcome this, OSUA is designed to be a hand-free, self-serve service where users only need to click a button (or submit a CLI command). Any failures caused by upgrades will be reported back quickly to the corresponding teams. To customize the upgrade processes for different teams, OSUA also allows users to set up and manage their own policy of upgrades.
Technical approach
At a high level, a server (as an example) needs to go through the following three steps sequentially for an OS upgrade:
Figure 1: General steps of hosts undergoing an OS upgrade
- Drain: Gracefully stopping or evacuating the applications that are running on the host serving traffic, sometimes with extra steps of initiating data balancing for stateful systems.
- Upgrade: A server can go through a full reimaging to upgrade images that have the main partition cleaned but the data partition retained or through a yum update way of reimaging that has both main and data partitions retained. At the end of an upgrade, all servers need to have machine health checks, depending on their hardware and system specs, performed and passed before serving any payloads after upgrade.
- Recover: Applications will be redeployed onto the server if they were allocated to it and not moved elsewhere as part of the drain step, possibly with data rebalancing and handling for stateful systems. For servers that had ephemeral applications evacuated, they become ready for any resource allocation of new workloads.
These three steps seem simple to be done quickly but are much more complex at scale. To upgrade the entire LinkedIn fleet and provide the features listed above, OSUA is built at the orchestration layer to manage and coordinate the upgrades with the following highlights.
Unified workflow
To support heterogeneous environments while maintaining common upgrade processes and experiences, after numerous internal researches, request gathering, and case studies, we’ve developed a single workflow that could be a one-fit-all solution with portability and flexibility for various applications and resource schedulers’ characteristics. Having one workflow, like OSUA, also helps to reduce onboarding and education efforts.
When a host/server/vm is the working unit in the upgrade process, the workflow can be shown as follows:
Figure 2: Unified workflow of steps of hosts undergoing an OS upgrade with optional pre-/post-step
Here are some design decisions worthwhile to highlight:
- Customized handling for drain and recover phases provide applications with the ability to handle necessary tasks before and after upgrades in a customized way. These abilities are essential to preparing stateful systems for an upgrade and recover back to the pre-upgrade ready-to-serve condition.
- Drain and recover phases are abstract. As they are tasks formatted as jobs encoded in rest.li schema for multiple consumers (resource scheduler in this case) to work on, any consumer can be plugged in and execute the tasks of its kind in a different way according to their own needs.
Impact analysis and batching
At LinkedIn, all OS upgrades are performed while live traffic is served. Therefore, during the drain phase, OSUA can only take down a computed subset of hosts that are submitted into its pipeline as a way of making use of capacity redundancy/reserve to ensure that linkedin.com has the needed capacity served.
OSUA leverages an internal standardized impact approval system (Blessin) that allows application teams to specify acceptable impact as a percentage of total number of instances / capacity as an absolute number, or consults customized built API, provided by individual service controllers (often cluster management services), to obtain information if a group of instances can be taken down and when.
While processing each host, OSUA figures out all of the application instances on the host and validates based on the configured rules in Blessin to determine if the application instances can be taken down. If all of the application instances on a host can be taken down, the host is picked for OS upgrade. The following figure illustrates a simplified example of determining if a host can be taken down for upgrades or if extra coordination, such as waiting, is needed.
Figure 3: Example of impact analysis process
Some stateful teams have a condition where a group of hosts within a fault zone (a logical group where all-or-none hosts in it can perform maintenance all at once) should be upgraded together so the overall cluster’s rebalances can be kept minimal. In such a scenario, OSUA tries to drain, upgrade, and recover such hosts as a single batch if all of the hosts in the batches are approved.
To maximize throughput, the impact analysis and batching mechanism is streamlined and conducted by intervals with parallelization to timely refresh data (such as capacity and upgrade status) and continue to pick hosts for upgrades.
Cross-system operation coordination
OSUA won’t be the only system doing maintenance on the site. There will be constant deployments taking place on application instances initiated by other maintenance activities such as data defragmentation, repartition on stateful systems, network switch upgrades, etc. Such maintenance activities, along with routine code release actions, have to be coordinated well so that, on a host, only one activity can take place at a time. Otherwise, OSUA could pick a host to drain and at the same time a routine code release could take place, which affects the cluster health of application instances too and consequently the ultimate impact in total would exceed the allowance.
Figure 4: Workflow of OSUA acquiring a lock from Insync while a system tries but fails to get the lock
To avoid this race condition, our SRE teams are working on a centralized locking system (Insync) where application instances and hosts can be locked for certain maintenance or release activities to ensure only one activity can take place at a time using a first in, first out (FIFO) method. A host that is locked successfully is considered down for maintenance in effective availability calculation during impact analysis. OSUA picks a host for maintenance only if the effective availability of each of the application instances is within the threshold configured by the owners of the application, and if the host is not locked already for any other maintenance activity.
Customized execution handling for stateful systems and more
While keeping the OS upgrade workflow unified that most applications can leverage, there are a number of systems that need customized handling in the format of pluggable add-on steps to the workflow because of their systematic complexities. One of the examples is the stateful system.
A stateful system is one where the operation of the system depends on a critical internal “state.” This state could be data or metadata that acts as the memory and history of the system at that point in time. The LinkedIn technical ecosystem comprises many stateful applications, especially on the data tier. These systems often have custom workflows that need to be executed before taking a node out of rotation (a.k.a. pre-steps) or bringing them back into the cluster (a.k.a. post-steps). These workflows vary quite a bit across the fleet and pose a bigger challenge for an automated OS upgrade setup.
In the past, engineers would need to run a number of administrative tasks manually or use scripts on to-be-upgraded hosts to ensure all necessary pre-steps are completed. Additionally, the problem is often compounded by the need to migrate data out of the to-be-upgraded host and rebalance the data across the rest of the cluster so that a minimum safe number of copies is maintained within the cluster. OSUA has to solve these diverse sets of problems while ensuring that no human toil is involved during the upgrade process.
To address the diverse demands from these systems, OSUA aligns towards a solution that is uniform in approach and still allows flexibility to these stateful systems to automate for their unique requirements for upgrades. As a result, OSUA leverages an in-house platform, STORU. STORU was initially developed with the idea of automating large scale operations for switch upgrades, but the system was extensible and supports customized automation before/after operations.
For pre-steps and post-steps, OSUA leverages a feature of STORU, custom hooks, which enables application owners to build custom application logic that would be executed before and after the OS upgrade process.
Figure 5: An example of custom hook execution of pre-step
In this section, we will focus on custom hooks and explore some of its salient features.
- Pre- and post-steps: As discussed earlier, a pre-step of the custom hook allows custom code execution to get hosts ready. This is usually required to safely take hosts out of rotation with optional customized extra steps. A post-step is a mirror image of the pre-step that is executed after the OS upgrade is complete to revert the outcomes of the pre-step.
- Custom hook execution order: OSUA allows custom hooks to be executed in various stages, which are defined relative to the application deployment step during the upgrade process. Both pre- and post-steps can be executed before, after, or both before and after application (un)deployment. This provides flexibility for stateful applications to configure how custom code execution can be invoked.
- Custom parameters: OSUA also allows application teams to define and pass additional parameters to custom hooks when submitting host(s) for upgrade. This helps custom code handle specific nuanced cases that might apply to a subset of hosts in the fleet when they are submitted for upgrade.
Auto-remediation
At scale, there will always be a certain percentage of failure that can occur during any steps involved in the OS upgrade process ranging from unsuccess of application uninstallation and deployment to server breakdown. OSUA is equipped with mechanisms to detect, analyze, triage, and remediate failures automatically, which greatly reduces human toil and facilitates company-wide hardware repair and refresh.
Self-contained
OSUA by nature is an infrastructure service. To be self-contained and avoid circular dependencies, which can result in cascading outages, we build OSUA on top of a limited number of internal control plane services and don’t depend on large scale data plane systems if we are able to find alternative solutions. For example, for event messaging needs, instead of using LinkedIn’s ready-for-use Kafka clusters, we implemented a lightweight restful based pub-sub mechanism within OSUA. This is to avoid the circular dependency such as Kafka uses Kafka (as a OSUA dependency) to upgrade host OS of Kafka that can lead to cascading failure when an upgrade is unsuccessful.
A recent LinkedIn OS upgrade
Since introducing OSUA, it has successfully performed more than 200k upgrades at LinkedIn, with more than seven million system packages updated and 18 million vulnerabilities addressed on these servers, with no external impact to LinkedIn customers and members from outages rooted from systematic processes. Further, the engineering effort from engineering teams to spend on OS upgrades has been reduced by 90% from previous upgrades that had an even smaller scale. The daily peak upgrade velocity is a 10x improvement from previous upgrades.
Now, many LinkedIn engineering teams come to this single platform to delegate OS upgrade operations worry-free.
Next steps
OSUA has shown success recently on LinkedIn’s on-prem infrastructure upgrade. However, increasing upgrade velocity with lower failure rate and less human intervention will be our continuous focus.
Acknowledgements
OSUA could not have been accomplished without the help of many engineers, managers, and TPMs across many teams. The engineers who have made contributions to OSUA are: Anil Alluri, Aman Sharma, Anant Bir Singh, Barak Zhou, Clint Joseph, Hari Prabhakaran, Jose Thomas Kiriyanthan, Junyuan Zeng, Keith Ward, Nikhita Kataria, Parvathy Geetha, Ronak Nathani, Ritu Panjwani, Subhas Sinha, Sagar Ippalpalli, Tim McNally, Vijay Bais, Ying He, Yash Shah, John Sushant Sundharam, and Deepshika. Special thanks to our TPMs Sean Patrick and Soumya Nair who have been steering this project from Day 1. Also, we’d like to thank the engineering leadership, Ashi Sareen, Mir Islam, Samir Tata, Sankar Hariharan, and Senthilkumar Eswaran, who have been providing continuous support to building OSUA. Additionally, we would like to thank Adam Debus, Justin Anderson, and Samir Jafferalifor their reviews and valuable feedback.
Topics
Since learning frontend and backend skills, Rishika’s passion for engineering has expanded beyond her team at LinkedIn to grow into her own digital community. As she develops as an engineer, giving back has become the most rewarding part of her role.
From intern to engineer—life at LinkedIn
My career with LinkedIn began with a college internship, where I got to dive into all things engineering. Even as a summer intern, I absorbed so much about frontend and backend engineering during my time here. When I considered joining LinkedIn full-time after graduation, I thought back to the work culture and how my manager treated me during my internship. Although I had a virtual experience during COVID-19, the LinkedIn team ensured I was involved in team meetings and discussions. That mentorship opportunity ultimately led me to accept an offer from LinkedIn over other offers.
Before joining LinkedIn full-time, I worked with Adobe as a Product Intern for six months, where my projects revolved around the core libraries in the C++ language. When I started my role here, I had to shift to using a different tech stack: Java for the backend and JavaScript framework for the frontend. This was a new challenge for me, but the learning curve was beneficial since I got hands-on exposure to pick up new things by myself. Also, I have had the chance to work with some of the finest engineers; learning from the people around me has been such a fulfilling experience. I would like to thank Sandeep and Yash for their constant support throughout my journey and for mentoring me since the very beginning of my journey with LinkedIn.
Currently, I’m working with the Trust team on building moderation tools for all our LinkedIn content while guaranteeing that we remove spam on our platform, which can negatively affect the LinkedIn member experience. Depending on the project, I work on both the backend and the frontend, since my team handles the full-stack development. At LinkedIn, I have had the opportunity to work on a diverse set of projects and handle them from end to end.
Mentoring the next generation of engineering graduates
I didn’t have a mentor during college, so I’m so passionate about helping college juniors find their way in engineering. When I first started out, I came from a biology background, so I was not aware of programming languages and how to translate them into building a technical resume. I wish there would have been someone to help me out with debugging and finding solutions, so it’s important to me to give back in that way.
I’m quite active in university communities, participating in student-led tech events like hackathons to help them get into tech and secure their first job in the industry. I also love virtual events like X (formally Twitter) and LinkedIn Live events. Additionally, I’m part of LinkedIn’s CoachIn Program, where we help with resume building and offer scholarships for women in tech.
Influencing online and off at LinkedIn
I love creating engineering content on LinkedIn, X, and other social media platforms, where people often contact me about opportunities at LinkedIn Engineering. It brings me so much satisfaction to tell others about our amazing company culture and connect with future grads.
When I embarked on my role during COVID-19, building an online presence helped me stay connected with what’s happening in the tech world. I began posting on X first, and once that community grew, I launched my YouTube channel to share beginner-level content on data structures and algorithms. My managers and peers at LinkedIn were so supportive, so I broadened my content to cover aspects like soft skills, student hackathons, resume building, and more. While this is in addition to my regular engineering duties, I truly enjoy sharing my insights with my audience of 60,000+ followers. And the enthusiasm from my team inspires me to keep going! I’m excited to see what the future holds for me at LinkedIn as an engineer and a resource for my community on the LinkedIn platform.
About Rishika
Rishika holds a Bachelor of Technology from Indira Gandhi Delhi Technical University for Women. Before joining LinkedIn, she interned at Google as part of the SPS program and as a Product Intern at Adobe. She currently works as a software engineer on LinkedIn’s Trust Team. Outside of work, Rishika loves to travel all over India and create digital art.
Editor’s note: Considering an engineering/tech career at LinkedIn? In this Career Stories series, you’ll hear first-hand from our engineers and technologists about real life at LinkedIn — including our meaningful work, collaborative culture, and transformational growth. For more on tech careers at LinkedIn, visit: lnkd.in/EngCareers.
Lekshmy has always been interested in a role in a company that would allow her to use her people skills and engineering background to help others. Working as a software engineer at various companies led her to hear about the company culture at LinkedIn. After some focused networking, Lekshmy landed her position at LinkedIn and has been continuing to excel ever since.
How did I get my job at LinkedIn? Through LinkedIn.
Before my current role, I had heard great things about the company and its culture. After hearing about InDays (Investment Days) and how LinkedIn supports its employees, I knew I wanted to work there.
While at the College of Engineering, Trivandrum (CET), I knew I wanted to pursue a career in software engineering. Engineering is something that I’m good at and absolutely love, and my passion for the field has only grown since joining LinkedIn. When I graduated from CET, I began working at Groupon as a software developer, starting on databases, REST APIs, application deployment, and data structures. From that role, I was able to advance into the position of software developer engineer 2, which enabled me to dive into other software languages, as well as the development of internal systems. That’s where I first began mentoring teammates and realized I loved teaching and helping others. It was around this time that I heard of LinkedIn through the grapevine.
Joining the LinkedIn community
Everything I heard about LinkedIn made me very interested in career opportunities there, but I didn’t have connections yet. I did some research and reached out to a talent acquisition manager on LinkedIn and created a connection which started a path to my first role at the company.
When I joined LinkedIn, I started on the LinkedIn Talent Solutions (LTS) team. It was a phenomenal way to start because not only did I enjoy the work, but the experience served as a proper introduction to the culture at LinkedIn. I started during the pandemic, which meant remote working, and eventually, as the world situation improved, we went hybrid. This is a great system for me; I have a wonderful blend of being in the office and working remotely. When I’m in the office, I like to catch up with my team by talking about movies or playing games, going beyond work topics, and getting to know each other. With LinkedIn’s culture, you really feel that sense of belonging and recognize that this is an environment where you can build lasting connections.
LinkedIn: a people-first company
If you haven’t been able to tell already, even though I mostly work with software, I truly am a people person. I just love being part of a community. At the height of the pandemic, I’ll admit I struggled with a bit of imposter syndrome and anxiety. But I wasn’t sure how to ask for help. I talked with my mentor at LinkedIn, and they recommended I use the Employee Assistance Program (EAP) that LinkedIn provides.
I was nervous about taking advantage of the program, but I am so happy that I did. The EAP helped me immensely when everything felt uncertain, and I truly felt that the company was on my side, giving me the space and resources to help relieve my stress. Now, when a colleague struggles with something similar, I recommend they consider the EAP, knowing firsthand how effective it is.
Building a path for others’ growth
With my mentor, I was also able to learn about and become a part of our Women in Technology (WIT) WIT Invest Program. WIT Invest is a program that provides opportunities like networking, mentorship check-ins, and executive coaching sessions. WIT Invest helped me adopt a daily growth mindset and find my own path as a mentor for college students. When mentoring, I aim to build trust and be open, allowing an authentic connection to form. The students I work with come to me for all kinds of guidance; it’s just one way I give back to the next generation and the wider LinkedIn community. Providing the kind of support my mentor gave me early on was a full-circle moment for me.
Working at LinkedIn is everything I thought it would be and more. I honestly wake up excited to work every day. In my three years here, I have learned so much, met new people, and engaged with new ideas, all of which have advanced my career and helped me support the professional development of my peers. I am so happy I took a leap of faith and messaged that talent acquisition manager on LinkedIn. To anyone thinking about applying to LinkedIn, go for it. Apply, send a message, and network—you never know what one connection can bring!
About Lekshmy
Based in Bengaluru, Karnataka, India, Lekshmy is a Senior Software Engineer on LinkedIn’s Hiring Platform Engineering team, focused on the Internal Mobility Project. Before joining LinkedIn, Lekshmy held various software engineering positions at Groupon and SDE 3. Lekshmy holds a degree in Computer Science from the College of Engineering, Trivandrum, and is a trained classical dancer. Outside of work, Lekshmy enjoys painting, gardening, and trying new hobbies that pique her interest.
Editor’s note: Considering an engineering/tech career at LinkedIn? In this Career Stories series, you’ll hear first-hand from our engineers and technologists about real life at LinkedIn — including our meaningful work, collaborative culture, and transformational growth. For more on tech careers at LinkedIn, visit: lnkd.in/EngCareers.
Topics
Espresso is the database that we designed to power our member profiles, feed, recommendations, and hundreds of other Linkedin applications that handle large amounts of data and need both high performance and reliability. As Espresso continued to expand in support of our 950M+ member base, the number of network connections that it needed began to drive scalability and resiliency challenges. To address these challenges, we migrated to HTTP/2. With the initial Netty based implementation, we observed a 45% degradation in throughput which we needed to analyze then correct.
In this post, we will explain how we solved these challenges and improved system performance. We will also delve into the various optimization efforts we employed on Espresso’s online operation section, implementing one approach that resulted in a 75% performance boost.
Espresso Architecture
Figure 1. Espresso System Overview
Figure 1 is a high-level overview of the Espresso ecosystem, which includes the online operation section of Espresso (the main focus of this blog post). This section comprises two major components – the router and the storage node. The router is responsible for directing the request to the relevant storage node and the storage layer’s primary responsibility is to get data from the MySQL database and present the response in the desired format to the member. Espresso utilizes the open-source framework Netty for the transport layer, which has been heavily customized for Espresso’s needs.
Need for new transport layer architecture
In the communication between the router and storage layer, our earlier approach involved utilizing HTTP/1.1, a protocol extensively employed for interactions between web servers and clients. However, HTTP/1.1 operates on a connection-per-request basis. In the context of large clusters, this approach led to millions of concurrent connections between the router and the storage nodes. This resulted in constraints on scalability, resiliency, and numerous performance-related hurdles.
Scalability: Scalability is a crucial aspect of any database system, and Espresso is no exception. In our recent cluster expansion, adding an additional 100 router nodes caused the memory usage to spike by around 2.5GB. The additional memory can be attributed to the new TCP network connections within the storage nodes. Consequently, we experienced a 15% latency increase due to an increase in garbage collection. The number of connections to storage nodes posed a significant challenge to scaling up the cluster, and we needed to address this to ensure seamless scalability.
Resiliency: In the event of network flaps and switch upgrades, the process of re-establishing thousands of connections from the router often breaches the connection limit on the storage node. This, in turn, causes errors and the router to fail to communicate with the storage nodes.
Performance: When using the HTTP/1.1 architecture, routers maintain a limited pool of connections to each storage node within the cluster. In some larger clusters, the wait time to acquire a connection can be as high as 15ms at the 95th percentile due to the limited pool. This delay can significantly affect the system’s response time.
We determined that all of the above limitations could be resolved by transitioning to HTTP/2, as it supports connection multiplexing and requires a significantly lower number of connections between the router and the storage node.
We explored various technologies for HTTP/2 implementation but due to the strong support from the open-source community and our familiarity with the framework, we went with Netty. When using Netty out of the box, the HTTP/2 implementation throughput was 45% less than the original (HTTP/1.1) implementation. Because the out of the box performance was very poor, we had to implement different optimizations to enhance performance.
The experiment was run on a production-like test cluster and the traffic is a combination of access patterns, which include read and write traffic. The results are as follows:
| Protocol | QPS | Single Read Latency (P99) | Multi-Read Latency (P99) |
| HTTP/1.1 | 9K | 7ms | 25ms |
| HTTP/2 | 5K (-45%) | 11ms (+57%) | 42ms (+68%) |
On the routing layer, after further analysis using flame graphs, major differences between the two protocols are shown in the following table.
| CPU overhead | HTTP/1.1 | HTTP/2 |
| Acquiring a connection and processing the request | 20% | 32% (+60%) |
| Encode/Decode HTTP request | 18% | 32% (+77%) |
Improvements to Request/Response Handling
Reusing the Stream Channel Pipeline
One of the core concepts of Netty is its ChannelPipeline. As seen in Figure 1, when the data is received from the socket, it is passed through the pipeline which processes the data. Channel Pipeline contains a list of Handlers, each working on a specific task.
Figure 2. Netty Pipeline
In the original HTTP/1.1 Netty pipeline, a set of 15-20 handlers was established when a connection was made, and this pipeline was reused for all subsequent requests served on the same connection.
However, in HTTP/2 Netty’s default implementation, a fresh pipeline is generated for each new stream or request. For instance, a multi-get request to a router with over 100 keys can often result in approximately 30 to 35 requests being sent to the storage node. Consequently, the router must initiate new pipelines for all 35 storage node requests. The process of creating and dismantling pipelines for each request involving a considerable number of handlers turned out to be notably resource-intensive in terms of memory utilization and garbage collection.
To address this concern, a forked version of Netty’s Http2MultiplexHandler has been developed to maintain a queue of local stream channels. As illustrated in Figure 2, on receiving a new request, the multiplex handler no longer generates a new pipeline. Instead, it retrieves a local channel from the queue and employs it to process the request. Subsequent to request completion, the channel is returned to the queue for future use. Through the reuse of existing channels, the creation and destruction of pipelines are minimized, leading to a reduction in memory strain and garbage collection.
Figure 3. Sequence diagram of stream channel reuse
Addressing uneven work distribution among Netty I/O threads
When a new connection is created, Netty assigns this connection to one of the 64 I/O threads. In Espresso, the number of I/O threads is equal to twice the number of cores present. The I/O thread associated with the connection is responsible for I/O and handling the request/response on the connection. Netty’s default implementation employs a rudimentary method for selecting an appropriate I/O thread out of the 64 available for a new channel. Our observation revealed that this approach leads to a significantly uneven distribution of workload among the I/O threads.
In a standard deployment, we observed that 20% of I/O threads were managing 50% of all the total connections/requests. To address this issue, we introduced a BalancedEventLoopGroup. This entity is designed to evenly distribute connections across all available worker threads. During channel registration, the BalancedEventLoopGroup iterates through the worker threads to ensure a more equitable allocation of workload
After this change, during registering of a channel, an event loop with the number of connections below the average is selected.
private EventLoop selectLoop() { int average = averageChannelsPerEventLoop(); EventLoop loop = next(); if (_eventLoopCount > 1 && isUnbalanced(loop, average)) { ArrayList list = new ArrayList<>(_eventLoopCount); _eventLoopGroup.forEach(eventExecutor -> list.add((EventLoop) eventExecutor)); Collections.shuffle(list, ThreadLocalRandom.current()); Iterator it = list.iterator(); do { loop = it.next(); } while (it.hasNext() && isUnbalanced(loop, average)); } return loop; } Reducing context switches when acquiring a connection
In the HTTP/2 implementation, each router maintains 10 connections to every storage node. These connections serve as communication pathways for the router I/O threads interfacing with the storage node. Previously, we utilized Netty’s FixedChannelPool implementation to oversee connection pools, handling tasks like acquiring, releasing, and establishing new connections.
However, the underlying queue within Netty’s implementation is not inherently thread-safe. To obtain a connection from the pool, the requesting worker thread must engage the I/O worker overseeing the pool. This process led to two context switches. To resolve this, we developed a derivative of the Netty pool implementation that employs a high-performance, thread-safe queue. Now, the task is executed by the requesting thread instead of a distinct I/O thread, effectively eliminating the need for context switches.
Improvements to SSL Performance
The following section describes various optimizations to improve the SSL performance.
Offloading DNS lookup and handshake to separate thread pool
During an SSL handshake, the DNS lookup procedure for resolving a hostname to an IP address functions as a blocking operation. Consequently, the I/O thread responsible for executing the handshake might be held up for the entirety of the DNS lookup process. This delay can result in request timeouts and other issues, especially when managing a substantial influx of incoming connections concurrently.
To tackle this concern, we developed an SSL initializer that conducts the DNS lookup on a different thread prior to initiating the handshake. This method involves passing the InetAddress, that contains both the IP address and hostname, to the SSL handshake procedure, effectively circumventing the need for a DNS lookup during the handshake.
Enabling Native SSL encryption/decryption
Java’s default built-in SSL implementation carries a significant performance overhead. Netty offers a JNI-based SSL engine that demonstrates exceptional efficiency in both CPU and memory utilization. Upon enabling OpenSSL within the storage layer, we observed a notable 10% reduction in latency. (The router layer already utilizes OpenSSL.)
To employ Netty Native SSL, one must include the pertinent Netty Native dependencies, as it interfaces with OpenSSL through the JNI (Java Native Interface). For more detailed information, please refer to https://netty.io/wiki/forked-tomcat-native.html.
Improvements to Encode/Decode performance
This section focuses on the performance improvements we made when converting bytes to Http objects and vice versa. Approximately 20% of our CPU cycles are spent on encode/decode bytes. Unlike a typical service, Espresso has very rich headers. Our HTTP/2 implementation involves wrapping the existing HTTP/1.1 pipeline with HTTP/2 functionality. While the HTTP/2 layer handles network communication, the core business logic resides within the HTTP/1.1 layer. Due to this, each incoming request required the conversion of HTTP/2 requests to HTTP/1.1 and vice versa, which resulted in high CPU usage, memory consumption, and garbage creation.
To improve performance, we have implemented a custom codec designed for efficient handling of HTTP headers. We introduced a new type of request class named Http1Request. This class effectively encapsulates an HTTP/2 request as an HTTP/1.1 by utilizing wrapped Http2 headers. The primary objective behind this approach is to avoid the expensive task of converting HTTP/1.1 headers to HTTP/2 and vice versa.
For example:
public class Http1Headers extends HttpHeaders { private final Http2Headers _headers; …. } And Operations such as get, set, and contains operate on the Http2Headers:
@Override public String get(String name) { return str(_headers.get(AsciiString.cached(name).toLowerCase()); } To make this possible, we developed a new codec that is essentially a clone of Netty’s Http2StreamFrameToHttpObjectCodec. This codec is designed to translate HTTP/2 StreamFrames to HTTP/1.1 requests/responses with minimal overhead. By using this new codec, we were able to significantly improve the performance of encode/decode operations and reduce the amount of garbage generated during the conversions.
Disabling HPACK Header Compression
HTTP/2 introduced a new header compression algorithm known as HPACK. It works by maintaining an index list or dictionaries on both the client and server. Instead of transmitting the complete string value, HPACK sends the associated index (integer) when transmitting a header. HPACK encompasses two key components:
-
Static Table – A dictionary comprising 61 commonly used headers.
-
Dynamic Table – This table retains the user-generated header information.
The Hpack header compression is tailored to scenarios where header contents remain relatively constant. But Espresso has very rich headers with stateful information such as timestamps, SCN, and so on. Unfortunately, HPACK didn’t align well with Espresso’s requirements.
Upon examining flame graphs, we observed a substantial stack dedicated to encoding/decoding dynamic tables. Consequently, we opted to disable dynamic header compression, leading to an approximate 3% enhancement in performance.
In Netty, this can be disabled using the following:
Http2FrameCodecBuilder.forClient() .initialSettings(Http2Settings.defaultSettings().headerTableSize(0));Results
Latency Improvements
| P99.9 Latency | HTTP/1.1 | HTTP/2 |
| Single Key Get | 20ms | 7ms (-66%) |
| Multi Key Get | 80ms | 20ms (-75%) |
We observed a 75% reduction in 99th and 99.9th percentile multi-read and read latencies, decreasing from 80ms to 20ms.
Figure 4. Latency reduction after HTTP/2
We observed similar latency reductions across the 90th percentile and higher.
Reduction in TCP connections
| HTTP/1.1 | HTTP/2 | |
| No of TCP Connections | 32 million | 3.9 million (-88%) |
We observed an 88% reduction in the number of connections required between routers and storage nodes in some of our largest clusters.
Figure 5. Total number of connections after HTTP/2
Reduction in Garbage Collection time
We observed a 75% reduction in garbage collection times for both young and old gen.
| GC | HTTP/1.1 | HTTP/2 |
| Young Gen | 2000 ms | 500ms (+75%) |
| Old Gen | 80 ms | 15 ms (+81%) |
Figure 6. Reduction in time for GC after HTTP/2
Waiting time to acquire a Storage Node connection
HTTP/2 eliminates the need to wait for a storage node connection by enabling multiplexing on a single TCP connection, which is a significant factor in reducing latency compared to HTTP/1.1.
| HTTP/1.1 | HTTP/2 | |
| Wait time in router to get a storage node connection | 11ms | 0.02ms (+99%) |
Figure 7. Reduction is wait time to get a connection after HTTP/2
Conclusion
Espresso has a large server fleet and is mission-critical to a number of LinkedIn applications. With HTTP/2 migration, we successfully solved Espresso’s scalability problems due to the huge number of TCP connections required between the router and the storage nodes. The new architecture also reduced the latencies by 75% and made Espresso more resilient.
Acknowledgments
I would like to thank my colleagues Antony Curtis, Yaoming Zhan, BinBing Hou, Wenqing Ding, Andy Mao, and Rahul Mehrotra who worked on this project. The project demanded a great deal of time and effort due to the complexity involved in optimizing the performance. I would like to thank Kamlakar Singh and Yun Sun for reviewing the blog and providing valuable feedback.
We would also like to thank our management Madhur Badal, Alok Dhariwal and Gayatri Penumetsa for their support and resources, which played a crucial role in the success of this project. Their encouragement and guidance helped the team overcome challenges and deliver the project on time.
Topics
WhatsApp, Instagram and Messenger to Get AI Assistants; Meta Shows Off Image Generation Tool Emu
Meta Smart Glasses in Collaboration With Ray-Ban Launched, Allows Hands-Free Livestreaming
![youtube-shorts-monetization-guide-[how-much-can-you-make?]](https://www.social-ping.com/wp-content/uploads/2023/09/135449-youtube-shorts-monetization-guide-how-much-can-you-make-80x80.png)
YouTube Shorts Monetization Guide [How Much Can You Make?]
WhatsApp Spotted Working on New Colours, Icons for Chat Interface on Android
5 B2B Social Media Marketing Tactics That Actually Work
Reddit Rolls Out Contributor Program, Offering Real Money for Gold and Karma
Enhancing Security and Developer Productivity: LinkedIn’s Journey with Implementing Content Security Policy
![top-social-media-manager-interview-questions-and-answers-[2023]](https://www.social-ping.com/wp-content/uploads/2023/09/130190-top-social-media-manager-interview-questions-and-answers-2023-80x80.png)
Top Social Media Manager Interview Questions and Answers [2023]
What Is FedRAMP, and Why Is It So Important?
How To Save Time and Avoid Issues With Facebook Auto Reply
What is Lemon8? TikTok’s Sister App Explained
Open Sourcing iris-message-processor
-
Uncategorized2 weeks agoCommunity Manager: Job Description & Key Responsibilities
-
LINKEDIN1 week agoCareer stories: Influencing engineering growth at LinkedIn
-
OTHER1 week agoWhatsApp iPad Support Spotted in Testing on Latest iOS Beta, Improved Group Calls Interface on Android
-
Uncategorized2 weeks agoSocial Media Intelligence: What It Is & Why You Need It
-
![how-to-create-a-social-media-report-[free-template-included]](https://www.social-ping.com/wp-content/uploads/2023/09/132170-how-to-create-a-social-media-report-free-template-included-400x240.png)
![how-to-create-a-social-media-report-[free-template-included]](https://www.social-ping.com/wp-content/uploads/2023/09/132170-how-to-create-a-social-media-report-free-template-included-80x80.png)
Uncategorized2 weeks agoHow to Create a Social Media Report [Free Template Included]
-
OTHER1 week agoYouTube Announces AI-Enabled Editing Products for Video Creators
-
Uncategorized2 weeks agoThe Complete Guide to Social Media Video Specs in 2023
-
Uncategorized7 days agoVanity Metrics: Definition & Examples for Marketing