After interning with us, Beatrix resonated with the culture and community she found at LinkedIn, and rejoined us post-undergrad. As she continues exploring her passion for frontend (UI) and accessibility engineering, she shares why launching her career with LinkedIn is one of the best decisions she has made.
From intern to engineer
In 2019, my career with LinkedIn started with a UI (frontend) engineering internship in the San Francisco office. I had done a bit of user experience (UX) work before, so I was excited that there was a specific frontend role for interns at LinkedIn. I learned a lot about frontend development and immersed myself in LinkedIn’s culture. After my internship, I felt like I had barely scratched the surface of all there was for me to learn at LinkedIn, and I was so happy to get a return offer as a UI engineer on the LinkedIn Marketing Solutions team after I graduated in 2020.
I had a bit of an untraditional path to engineering — I’ve always been creative and loved taking Latin, but I started taking an interest in computer science during high school, which continued into my undergraduate studies at Vassar. Computer science combined many things I liked about science with my love for humanities and logic; it’s truly a multidisciplinary field.
I honed these skills during my extracurriculars in college as a teaching assistant with Kode with Klossy, a summer camp that helps teach teen girls how to code, and through attending the Grace Hopper Celebration, a women in tech conference. It was a full-circle moment when I had the privilege to attend the virtual Grace Hopper Conference in 2020 with LinkedIn.
A culture of Next Plays
One of the attributes that kept me here is LinkedIn’s culture of transformation. With every team I’ve been on, there’s a lot of celebration of people’s “Next Plays” as we call it here, whether that’s a new job opportunity or promotion at LinkedIn itself or elsewhere.
While I enjoyed my time working on LinkedIn Marketing Solutions’ Campaign Manager, after 1.5 years on the team, I was eager to dive deeper into a new challenge and more accessibility work, to better support diverse learners and LinkedIn users (or members as we call them) with disabilities.
Thanks to LinkedIn’s wonderful culture that has an emphasis on collaboration and mentorship, I was able to connect with engineers from across the engineering organization to find a role that combined my interests in accessibility engineering, and development for the main LinkedIn site. With that mentorship, I found a new role earlier this year on our LinkedIn Talent Solutions team, centered on the job search and evaluation engineering work.
I’ve always found LinkedIn to be very human in its approach to work, because everything we do stems from our mission to build economic opportunities and connections for people. The job search team is focused on trying to help people get jobs and with our accessibility impact, we make more jobs accessible to every member of the global workforce. This focus was also seen in other roles that I’ve had at LinkedIn. While on the LMS team, I got to work on our reflow efforts, which ensures that the pages in Campaign Manager are usable on many screen sizes and at different zoomed-in levels.
50M job seekers visit LinkedIn every week — resulting in 95 job applications every second, and six hires every minute. To help job seekers find their dream jobs at that scale is incredibly rewarding, and I feel very fortunate to contribute directly to those outcomes as an engineer.
Being there through the tough times
And LinkedIn’s human approach transcends the work itself. I grew up in Los Angeles, and I’m incredibly close to my parents, and two sisters: one who’s in high school, and one in college.
I remember getting pulled into a family emergency where I had to unexpectedly fly back home from San Francisco to help, and told my manager, “I’m not sure when I’ll be back [in San Francisco].” My team and managers were incredibly compassionate, and I was able to spend time with my family and fly back to visit them as needed to help support during this difficult time.
I was also able to be in Los Angeles from Thanksgiving to New Years with my family, working remotely from LA. The flexibility and earnest support I’ve received from my team during both the good and the tough times have meant the world.
Craftsmanship in engineering
On the technical side of the house, one of the things that impresses me about LinkedIn is engineering’s emphasis on craftsmanship here, especially on our Talent Solutions team. We invest a lot into the foundations of our code base and code quality, ensuring that we are writing code that we can build on in the future.
While working on new features for the site is always exciting, I am also grateful to have the opportunity to work on the efforts that improve the site behind the scenes, like documentation and other quality-of-life changes. Many teams at LinkedIn are trying to push foundational work initiatives like this forward. Documentation is one of those things that always comes up in developer productivity and happiness here at LinkedIn, so I’m glad to be able to contribute in ways that help make my colleagues’ work lives easier.
Recently, my team discussed a situation with a contractor who came into the code base and thought our code tests did not make sense. This confusion sparked us to begin renaming the tests, changing the wording, and agreeing on the clearest way of labeling our code tests. I have so appreciated having space for these discussions; although product users will never see this, it is something that makes our code so much more reliable.
Breaking barriers through Women in Tech
Since I joined LinkedIn full-time in the midst of remote work during the pandemic, I wanted to find ways to connect with other engineers. I’m so thankful that LinkedIn has given me those opportunities; I was one of the founding members of the LinkedIn Marketing Solutions branch of Women in Tech (LMS WiT), and joined our Out@In (i.e., LGBTQIA+) Employee Resource Group (ERG). It is incredible how leadership opportunities aren’t gated to you based on age or company tenure at LinkedIn. I was able to grow and learn so much about what it means to be organized, to be a leader, and what it means to think about how I am in a position to help the WIT community, to facilitate these learnings.
Within LMS WiT, I helped to co-found the Amplify Voices track. Shortly after joining, I raised that we should rename our Male Allies track. I had heard from several nonbinary employees on our LinkedIn Marketing Solutions team that were wondering if there was room for them within WiT. It was powerful to me that my group was receptive to my idea and changed the name to WiT Allies the very next day, so that more LinkedIn employees felt included. If you’re interested in equality, empowerment, and these events that are focused on how to speak up for yourself in a professional setting, it’s essential to have these discussions about inclusiveness.
Anytime I had a suggestion in ERGs, it was always considered thoughtfully and there was a lot of trust placed in me even as a young professional. In LinkedIn’s ERGs, there’s this openness that breaks down artificial limits and helps us grow as leaders. This spirit of inclusiveness is what makes LinkedIn such a welcoming place.
Beatrix is a frontend (UI) engineer on our LinkedIn Talent Solutions team. Prior to her current role, Beatrix was a UI engineering intern and a UI engineer on our LinkedIn Marketing Solutions team. She graduated from Vassar College with a degree in computer science. In her free time, Beatrix enjoys spending time with her two cats, Mr. Darcy and Georgiana, cross-stitching and crocheting, and gaming.
Editor’s note: Considering an engineering/tech career at LinkedIn? In this Career Stories series, you’ll hear first-hand from our engineers and technologists about real life at LinkedIn — including our meaningful work, collaborative culture, and transformational growth. For more on tech careers at LinkedIn, visit: lnkd.in/EngCareers.
Real-time analytics on network flow data with Apache Pinot
The LinkedIn infrastructure has thousands of services serving millions of queries per second. At this scale, having tools that provide observability into the LinkedIn infrastructure is imperative to ensure that issues in our infrastructure are quickly detected, diagnosed, and remediated. This level of visibility helps prevent the occurrence of outages so we can deliver the best experience for our members. To provide observability, there are various data points that need to be collected, such as metrics, events, logs, and flows. Once collected, the data points can then be processed and made available, in real-time, for engineers to use for alerting, troubleshooting, capacity planning, and other operations.
At LinkedIn, we developed InFlow to provide observability into network flows. A network flow describes the movement of a packet through a network and is the metadata of a packet sampled at a network device that describes the packet in terms of the 5-tuple: source IP, source port, destination IP, destination port, and protocol. It may also contain source and destination autonomous system numbers (ASNs), the IP address of the network device that has captured this flow, input and output interface indices of the network device where the traffic was sampled, and the number of bytes transferred.
How LinkedIn leverages flow data
InFlow provides a rich set of time-series network data having over 50 dimensions such as source and destination sites, security zones, ASNs, IP address type, and protocol. With this data, various types of analytical queries can be run to get meaningful insights about network health and characteristics.
Figure 1. A screenshot from InFlow UI’s Top Services tab which shows the 5 services consuming the most network bandwidth and the variation of this traffic over the last 2 hours
Most commonly, InFlow is used for operational troubleshooting to get complete visibility into the traffic. For example, if there is an outage due to a network link capacity exhaustion, InFlow can be used to find out the top talkers for that link based on hosts/services that are consuming the most bandwidth (Figure 1) and based on the nature of the service, further steps can be taken to remediate the issue.
Flow data also provides source and destination ASN information, which can be used for optimizing cost, based on bandwidth consumption of different kinds of peering with external networks. It can also be used for analyzing data based on several dimensions for network operations. For example, finding the distribution of traffic by IPv4 or IPv6 flows or the distribution of traffic based on Type of Service (ToS) bits.
InFlow architecture overview
Figure 2. InFlow architecture
Figure 2 shows the overall InFlow architecture. The platform is divided into 3 main components: flow collector, flow enricher, and InFlow API with Pinot as a storage system. Each component has been modeled as an independent microservice to provide the following benefits:
- It enforces the single responsibility principle and prevents the system from becoming a monolith.
- Each of the components have different requirements in terms of scaling. Separate microservices ensure that each can be scaled independently.
- This architecture creates loosely coupled pluggable services which can be reused for other scenarios.
InFlow receives 50k flows per second from over 100 different network devices on the LinkedIn backbone and edge devices. InFlow supports sFlow and IPFIX as protocols for collecting flows from network devices. This is based on the device’s vendor support for the protocols and minimal impact of flow export on the device’s performance. The InFlow collector receives and parses these incoming flows, aggregates the data into unique flows for a minute, and pushes them to a Kafka topic for raw flows.
The data processing pipeline for InFlow leverages Apache Kafka and Apache Samza for stream processing of incoming flow events. Our streaming pipeline processes 50k messages per second, enriching the data with 40 additional fields (like service, source and destination sites, security zones, ASNs, and IP address type), which are fetched from various internal services at LinkedIn. For example, our data center infrastructure management system, InOps, provides information on the site, security zone, security domain of the source, and destination IPs for a flow. The incoming raw flow messages are consumed by a stream processing job on Samza and after adding the additional enriched fields, the result is pushed to an enriched Kafka topic.
InFlow requires storage of tens of TBs of data with a retention of 30 days. To support its real-time troubleshooting use case, the data must be queryable in real-time with sub-second latency so that engineers can query the data without any hassles during outages. For the storage layer, InFlow leverages Apache Pinot.
Figure 3. A screenshot from InFlow UI’s Explore tab which provides a self-service interface for users to visualize flow data by grouping and filtering on different dimensions
The InFlow UI is a dashboard with some of the commonly used visualizations on flow data pre-populated that provides a rich interface where the data can be filtered or grouped by any of the 40 different dimension fields. The UI also has an Explore section, which allows for creation of ad-hoc queries. The UI is based on top of InFlow API, which is a middleware responsible for translating user input into Pinot queries and issuing them to the Pinot cluster.
Pinot as a storage layer
In the first version of InFlow, data was ingested from the enriched Kafka topic to HDFS. We leveraged Trino for facilitating user queries on the data present in HDFS. However, the ETL and aggregation pipeline added a 15-20 minute delay to the pipeline, reducing the freshness of the data. Additionally, query latencies to HDFS using Presto were in the order of 15-30 seconds. This latency and delay was acceptable for doing historical data analytics, however, for real-time troubleshooting, the data needs to be available in real-time with a maximum delay of 1 minute.
Based on the query latency and data freshness requirements, we explored several storage solutions available at LinkedIn (like Espresso, Kusto, and Pinot) and decided on onboarding our data to Apache Pinot. When looking for solutions, we needed a reliable system providing real-time ingestion and sub-second query latencies. Pinot’s support for Lambda and Lamda-less architecture, real-time ingestion, and low latency at high throughput could help us achieve optimal results. Additionally, the Pinot team at LinkedIn is experimenting with supporting a new use case called Real-time Operational Metrics Analysis (ROMA), which enables engineers to slice and dice metrics along different combinations of dimensions to help monitor infrastructure near real-time, analyze the last few weeks/months/years of data to discover trends and patterns to forecast and plan capacity, and helps to find the root cause of outages quickly and reduce the time to recovery. These objectives aligned well with our problem statement of processing large numbers of metrics in real-time.
The Pinot ingestion pipeline consumes directly from the enriched Kafka topic and creates the segments on the Pinot servers, which improves the freshness of the data in the system to less than a minute. User requests from InFlow UI are converted to Pinot SQL queries and sent to the Pinot broker for processing. Since Pinot servers keep data and indices in cache-friendly data structures, the query latencies are a huge improvement from the previous version where data was queried from disk (HDFS).
Several optimizations were done to reach this query latency and ingestion parameters. Because the data volume for the input Kafka topic is high, several considerations were made to decide the optimal number of partitions in the topic to allow for parallel consumption into segments in Pinot after several experiments with the ingestion parameters. Most of our queries involved a regexp_like condition on the devicehostname column, which is the name of the network device that has exported the flow. This is used to narrow down on a specific plane of the network. regexp_like is inefficient as it cannot leverage any index so to resolve this, we set up an ingestion transformation using Pinot. These are various transformation functions that can be applied to your data before it is ingested into Pinot. The transformation created a derived column flowType, which classifies a flow based on the name of the network device that has exported this flow into a specific plane of the network. For example, if the exporting device is at the edge of our network, then this flow can be classified as an Internet-facing flow. The flowType column is now an indexed column used for equality comparisons instead of regexp_like and this helped improve query latency by 50%.
Queries from InFlow always request for data from a specific range in time. To improve query performance, timestamp based pruning was enabled on Pinot. This improved query latencies since only relevant segments are filtered in for processing based on the filter conditions on the timestamp column in queries. Based on the Pinot team’s input, indexes on the different dimension columns were set up to aid query performance.
Figure 4. Latency metric for InFlow API query for top flows in the last 12 hours before and after onboarding to Pinot
Following the successful onboarding of flow data to a real-time table on Pinot, freshness of data improved from 15 mins to 1 minute and query latencies were reduced by as much as 95%. For some of the more expensive queries, which took as much as 6 minutes using Presto queries, the query latency reduced to 4 seconds using Pinot.This has been helpful in making it easier for the network engineers at LinkedIn to easily get the data they need for troubleshooting or running real-time analytics on network flow data.
The current network flow data only provides us with sampled flows from the LinkedIn backbone and edge network. Skyfall is an eBPF-based agent, developed at LinkedIn, that collects flow data and network metrics from the host’s kernel with minimal overhead. The agent captures all flows for the host without sampling and will be deployed across all servers in the LinkedIn fleet. This would provide us with a 100% coverage of flows across our data centers and enable us to support more use cases on flow data that require unsampled information such as security audit and validation based use cases. Because the agent collects more data and from more devices, the scale of data collected by Skyfall is expected to be 100 times that of InFlow. We are looking forward to leveraging the InFlow architecture to support this scale and provide real-time analytics on top of the rich set of metrics exported by the Skyfall agent. Another upcoming feature that we are excited about is leveraging InFlow data for anomaly detection and more traffic analytics.
Onboarding our data to Pinot was a collaborative effort and we would like to express our gratitude to Subbu Subramaniam, Sajjad Moradi, Florence Zhang, and the Pinot team at LinkedIn for their patience and efforts in understanding our requirements and working on the optimizations required for getting us to the optimal performance.
Thanks to Prashanth Kumar for the continuous dialogue in helping us understand the network engineering perspective on flow data. Thanks to Varoun P and Vishwa Mohan for their leadership and continued support.
Feathr joins LF AI & Data Foundation
In April 2022, Feathr was released under the Apache 2.0 license and we announced, in close conjunction with our Microsoft Azure partners, native integration and support for Feathr on Azure. Since being open sourced, Feathr has achieved substantial popularity among the machine learning operations (MLOps) community. It has been adopted by companies of various sizes across multiple industries and the community continues to grow rapidly. Most excitingly, more and more open-source enthusiasts are contributing code to Feathr.
It’s clear that many others experience the same pain points that Feathr aims to address. That’s why we are excited to share it with a broader audience and for Feathr to be adopted by a broader open-source community with help from LF AI & Data.
Donating to the LF AI & Data will help ensure that Feathr continues to grow and evolve across various dimensions, including visibility, user base, and contributor base. Also, the Feathr development team will have more opportunities to collaborate with other member companies and projects, such as achieving richer online store support via integration with Milvus and JanusGraph, and adopting open data lineage standard from OpenLineage. As a result, we hope Feathr helps AI engineers build and scale feature pipelines and feature applications in ways that push MLOps tech stacks and the industry forward for years to come.
The Feathr feature store provides an abstraction layer between raw data and ML models. This abstraction layer standardizes and simplifies feature definition, transformation, serving, storage, and access from within ML workflows or applications. Feathr empowers AI engineers to focus on feature engineering while it takes care of data serialization format, connecting to various databases, performance optimization, and credential management. More specifically, Feathr helps:
- Define features once and use them in different scenarios, like model training and model serving
- Create training dataset with point-in-time correct semantics
- Connect to various offline data sources (data lakes, and data warehouses), and then transform source data into features
- Deliver feature data from offline system to online store for faster online serving
- Discover features or share features among colleagues or teams with ease
To learn more please visit Feathr’s GitHub page here and our April 2022 blog, Open sourcing Feathr – LinkedIn’s feature store for productive machine learning.
Acceptance into the LF AI & Data indicates an important recognition from the Linux Foundation. We believe a large, diverse, healthy, and self-sustained Feathr open-source community is important. We’re excited for the new chapter of Feathr and to welcome more people into the Feathr community.
Career stories: Rejoining LinkedIn to scale our media infrastructure
Originally from Argentina, systems & infrastructure engineering leader Federico was a founding member of the Media Infrastructure team in 2015. Now based in Bellevue, Wash., Federico shares how his supportive mentor, LinkedIn’s “sweet spot” scale, and the distinctive engineering challenges here ultimately brought him back to LinkedIn in 2019.
My love for engineering started in my home country of Argentina. After working as an engineer in a corporate setting for a few years, I decided to start my own company focused on custom software development. I loved the interesting problems I could solve every day for my clients, but I was searching for greater economic opportunities in the U.S., where most of my clients were based. After working as a contractor for YouTube, I found my passion for media and engineering of video systems.
Joining and rejoining LinkedIn
When LinkedIn reached out to me with an opportunity to build their video platform in 2015, I jumped at the chance. It was thrilling to join LinkedIn at a time when we launching in-feed video. What originally started as a team of two grew to nine people, and that’s when LinkedIn began training me to step into my first management role for the Media Infrastructure team.
After growing in my management position for a few years, I left LinkedIn for an opportunity working on larger scale systems. But I quickly became burned out and missed my original role as an individual contributor at LinkedIn. My previous manager at LinkedIn was so supportive. I was offered a role as a technical architect (i.e., Senior Staff) for media infrastructure, which allowed me to return to LinkedIn with new technical knowledge, and the same passion for my work.
Making the move to a new LinkedIn home base
Once our team had grown to almost 40 people, we reached the point at which it made sense to look for additional engineering talent outside the San Francisco Bay and New York City areas. It is challenging to find engineers in the media domain since very few companies are doing what LinkedIn does at scale. That’s when we started considering the next office location as an opportunity to bring in more talent.
Ultimately, we decided on Bellevue, Washington. After eight years in the Bay Area, I was ready for a move, and Bellevue was the right fit for my wife and me for many reasons. For example, many of the media companies we partnered with had a strong engineering presence in Seattle. Our driving motivation was to spearhead the company culture and to build an identity for a new LinkedIn office. The Bellevue office just turned one year old and we have been able to build a thriving engineering community here that’s growing quickly.
Taking ownership and giving back
In my current role as a Principal Staff Software Engineer, I love that I can mix the technical side of engineering with driving the strategic and product roadmap for my organization.
As an infrastructure engineer, there’s a sweet spot here between the scale of your work and the size of your engineering team at LinkedIn. We have relatively small teams tackling very large problems in complex technical domains. This creates great opportunities for individual ownership over a significant engineering problem on a large scale. We have space to get involved and truly make a difference instead of simply being a cog in a wheel.
Throughout my time in Silicon Valley, so many mentors were instrumental in shaping my career. As I’ve grown, I’ve tried to prioritize paying it forward by mentoring my team and other engineers at LinkedIn. Relationships matter, especially at LinkedIn. Building your network is a really core value here, because we thrive on connections.
More About Federico
Based in Bellevue, Washington, Federico is a Principal Staff Systems & Infrastructure Engineer on LinkedIn Media Infrastructure team. Prior to his time at LinkedIn, Federico’s engineering career led him from launching his own software development company, ESTUDIO42, to software engineering roles at YouTube and Instagram. Federico holds a degree in Computer Engineering from the Universidad Nacional de Tucuman in Argentina. Outside of work, Federico enjoys traveling with his wife, cooking, visiting shuttle expeditions, and mixing music.
How Brands Can Support Indigenous Communities on Social—the Right Way
US Senate Panel Approves Bill Empowering News Organisations to Negotiate With Facebook, Google for Revenue
WhatsApp Working to Keep Iranians Connected Amid Widespread Internet Shutdown Over Nationwide Protests
Facebook Whistleblower Frances Haugen Launches ‘Beyond the Screen’ Organisation to Tackle Social Media Harms
How To Win at TikTok (According to TikTok)
Meta for Business: How To Get the Best Results From Each Platform
Twitter Expanding Birdwatch Community Fact-Checking Programme With New Onboarding Process, More
Twitter Ordered to Give Information to Elon Musk Regarding Spam and Bot Accounts
Operating system upgrades at LinkedIn’s scale
Career stories: Rejoining LinkedIn to scale our media infrastructure
14 Best Apps For Instagram Collages in 2022
How to Write Your Best Social Media Bio [+28 Free Templates]
LINKEDIN2 weeks ago
Real-time analytics on network flow data with Apache Pinot
FACEBOOK1 week ago
Introducing Facebook Graph API v15.0 and Marketing API v15.0
FACEBOOK2 weeks ago
Meet the Developers: Linux Kernel Team (Jonathan Zhang)
Uncategorized1 week ago
The 12 Best Chatbot Examples for Businesses
LINKEDIN2 weeks ago
Feathr joins LF AI & Data Foundation
Uncategorized1 week ago
I Tried Instagram Automation (So You Don’t Have To): An Experiment
FACEBOOK4 days ago
Summer of open source: building more efficient AI with PyTorch
OTHER1 week ago
Zoom Resolves Connectivity Issues After Over 40,000 Users Reported Problem