Connect with us


Hermit: Deterministic Linux for Controlled Testing and Software Bug-finding




If you’ve used emulators for older platforms, you probably experienced a level of precise control over software execution that we lack on contemporary platforms. For example, if you play 8-bit video games emulated on your modern console, you are able to suspend and rewind gameplay, and when you resume, that incoming creature or projectile will predictably appear in the same spot because any randomness plays out deterministically within the emulator.

Yet, as a software engineer, when your multithreaded service crashes under transient conditions, or when your test is flaky, you don’t have these luxuries. Everything the processor and operating system contributes to your program’s execution—thread scheduling, random numbers, virtual memory addresses, unique identifiers—constitutes an unrepeatable, unique set of implicit inputs. Standard test-driven methodologies control for explicit program inputs, but they don’t attempt to control these implicit ones.

Since 2020, our team within DevInfra has worked to tackle this hard problem at its root: the pervasive nondeterminism in the interface between applications and the operating system. We’ve built the first practical deterministic operating system called Hermit (see endnote on prior art). Hermit is not a new kernel—instead it’s an emulation layer on top of the Linux kernel. In the same way that Wine translates Windows system calls to POSIX ones, Hermit intercepts system calls and translates them from the Deterministic Linux abstraction to the underlying vanilla Linux OS.

Details on sources of and solutions for nondeterminism can be found in our paper, “Reproducible Containers,” published in ASPLOS ’20, which showcased an earlier version of our system. We’ve open-sourced the new Hermit system and the underlying program-instrumentation framework named Reverie.

Example Applications

Now we explore some of the applications Hermit can be used for, and the role [non]determinism plays. In the next section, we go deeper into how Hermit works.

free widgets for website

Flaky tests

First, flaky tests. They’re a problem for every company. Google, Apple, Microsoft and Meta have all published their experiences with flaky tests at scale. Fundamentally, the cause of flakiness is that test functions don’t really have the signatures that appear in the source code. An engineer might think they’re testing a function from a certain input type to output type, for example:

 test : Fn(Input) -> Output; 

Indeed, unless we’re doing property-based testing, then for unit tests it’s even simpler. (The input is empty, and the output is boolean.) Unfortunately, in reality, most tests may be affected by system conditions and even external network interactions, so test functions have a true signature more like the following:

 test : Fn(Input, ThreadSchedule, RNG, NetworkResponses) -> Output; 

The problem is that most of these parameters are outside of engineers’ control. The test harness and test code, running on a host machine, is at the mercy of the operating system and any external services.

Caption: Irreproducible, implicit inputs from the operating system can affect test outcomes.

That’s where Hermit comes in. Hermit’s job is to create a containerized software environment where every one of the implicit inputs (pictured above) is a repeatable function of the container state or the container configuration, including command line flags. For example, when the application requests the time, we provide a deterministic time that is a function of program progress only. When an application thread blocks on I/O, it resumes at a deterministic point relative to other threads.

free widgets for website

Hermit’s guarantee is that any program run by Hermit (without external networking) runs an identical execution—irrespective of the time and place it is run—yielding an identical stream of instructions and complete memory states at the time of each instruction. This means if you run your network-free regression test under Hermit, it is guaranteed not to be flaky:

 hermit run ./testprog 

Further, Hermit allows us to not merely explore a single repeatable execution of a program, but to systematically navigate the landscape of possible executions. Let’s look at how to control one specific feature: pseudo-random number generation (PRNG). Of course, for determinism, when the application requests random bytes from the operating system, we provide repeatable pseudo-random ones. To run a program with different PRNG seeds, we simply use a different --rng-seed parameter:

 hermit run --rng-seed=1 prog hermit run --rng-seed=2 prog 

In this case, it doesn’t matter what language prog is written in, or what random number generator library it uses, it must ultimately ask the operating system for entropy, at which point it will receive repeatable pseudo-random inputs.

See also  Thomasville Fire Rescue Launches Official Facebook Page

It is the same for thread scheduling: Hermit takes a command line seed to control thread interleaving. Hermit is unique in being able to reproducibly generate schedules for full Linux programs, not merely record ones that happen in nature. It generates schedules via established randomized strategies, designed to exercise concurrency bugs. Alternatively, full thread schedules can be passed explicitly in as input files, which can be derived by capturing and modifying a schedule from a previous run. We’ll return to this in the next section.

There is an upshot to making all these implicit influences explicit. Engineers dealing with flaky programs can steer execution as they desire, to achieve the following goals:

free widgets for website
  • Regression testing: Use settings that keep the test passing.
  • Stress testing: Randomize settings to find bugs more effectively.
  • Diagnosis: Vary inputs systematically to find which implicit inputs cause flakiness.

Pinpointing a general class of concurrency bugs

As mentioned above, we can vary inputs to find what triggers a failure, and we can control schedules explicitly. Hermit builds on these basic capabilities to analyze a concurrency bug and pinpoint the root cause. First, Hermit searches through random schedules, similar to rr chaos mode. Once Hermit finds failing and passing schedules, it can analyze the schedules further to identify pairs of critical events that run in parallel and where flipping the order of those events causes the program to fail. These bugs are sometimes called ordering violations or race conditions (including but not limited to data races).

Many engineers use race detectors such as ThreadSanitizer or go run -race. Normally, however, a race detector requires compile-time support, is language-specific and works only to detect data races, specifically data races on memory (not files, pipes, etc.). What if, instead, we have a race between a client program written in Python, connecting to a server written in C++, where the client connects before the server has bound the socket? This nondeterministic failure is an instance of the “Async Await” flakiness category, as defined by an empirical analysis and classification of flaky tests.

By building on a deterministic operating system abstraction, we can explicitly vary the schedule to empirically find those critical events and print their stack traces. We start by using randomized scheduling approaches to generate a cloud of samples within the space of possible schedules:

Caption: A visualization of the (exponential) space of possible thread schedules, generated by different Hermit seeds. With the space organized by edit distance, the closest red and green dots correspond to the minimum edit distance observed between a passing and failing schedule.

We can organize this space by treating the thread schedules literally as strings, representing sequential scheduling histories. For example, with two threads A & B, “AABBA” could be an event history. The distance between points is the edit distance between strings (actually, a weighted edit distance preferring swaps over insertion or deletion). We can then take the closest pair of passing and failing schedules and then study it further. In particular, we can bisect between those schedules, following the minimum edit distance path between them, as visualized below.

See also  Facebook buys studio behind collaborative game-making platform 'Crayta'

Caption: A binary search between passing and failing schedules, probing points in between until it finds adjacent schedules that differ by a single transposition, a Damerau-Levenshtein distance of one.

free widgets for website

At this point, we’ve reduced the bug to adjacent events in the thread schedule, where flipping their order makes the difference between passing and failing. We report the stack traces of these events as a cause of flakiness. (Indeed, it is only a single cause because there may be others if flakiness is overdetermined.)

Challenges and how it works

Here, we’ll cover a bit about how Hermit works, emphasizing pieces that are different from our prototype from ASPLOS ’20. The basic scenario is the same, which is that we set out to build a user space determinization layer, not allowing ourselves the liberty of modifying the Linux kernel or using any privileged instructions.

Challenge 1: Interposing between operating system and application

Unfortunately, on Linux there is not a standard, efficient, complete method to interpose between user-space applications and the operating system. So we’ve built a completely new program sandboxing framework in Rust, called Reverie, that abstracts away the backend—how program sandboxing is implemented. Reverie provides a high-level Rust API to register handlers: callbacks on the desired events. The user writes a Reverie tool that observes guest events and maintains its own state.

Reverie is not just for observing events. When you write a Reverie handler, you are writing a snippet of operating system code. You intercept a syscall, and you become the operating system, updating the guest and tool state as you like, injecting zero or more system calls to the underlying Linux operating system and finally returning control to the guest. These handlers are async, and they run on the popular Rust tokio framework, interleaving with each other as multiple guest threads block and continue.

The reference Reverie backend uses the ptrace system call for guest event interception, and a more advanced backend uses in-memory program instrumentation. In fact, Reverie is the only program instrumentation library that abstracts away whether instrumentation code is running in a central place (its own process), or inside the guest processes themselves via injected code.

free widgets for website

Challenge 2: Inter-thread synchronization

Consider communication through sockets and pipes within the reproducible container. This is an area where our earlier prototype mainly used the strategy of converting blocking operations to non-blocking ones, and then polling them at deterministic points in time within a sequential execution. Because we run in user space, we don’t have a direct way to ask the kernel whether a blocking operation has completed, so attempting a non-blocking version of the syscall serves as our polling mechanism.

Hermit builds on this strategy and includes a sophisticated scheduler with thread priorities and multiple randomization points for exploring “chaos mode” paths through the code. This same scheduler implements a back-off strategy for polling operations to make it more efficient.

Hermit also goes beyond polling, implementing some inter-thread communication entirely inside Hermit. By including features like a built-in futex implementation, Hermit takes a small step closer to behaving like an operating system kernel. But Hermit is still vastly simpler than Linux and passes most of the heavy lifting on to Linux itself.

See also  Facebook made its bed and wants Congress to lie in it

For specific features that Hermit implements directly, it never passes those system calls through to Linux. In the case of futexes, for example, it is hard or impossible to come up with a series of raw futex syscalls to issue to the kernel and achieve a deterministic outcome. Subtleties include spurious wake-ups (without the futex value changing), the nondeterministic selection of threads to wake, and the ineradicable moment in time after Hermit issues a blocking syscall to Linux, but before we know for sure if Linux has acted on it.

These issues are avoided entirely by intercepting each futex call and updating the Hermit scheduler’s own state precisely and deterministically. The underlying Linux scheduler still runs everything, physically, but Hermit’s own scheduler takes precedence, deciding which thread to unblock next.

free widgets for website

Challenge 3: Large and complex binaries

Meta has no shortage of large and challenging binaries that use recent features of the Linux kernel. Nevertheless, after a couple of years of development, Hermit runs thousands of different programs deterministically today. This includes more than 90 percent of test binaries we run it on.

Other applications

While flaky tests and concurrency bugs are where we’ve invested most heavily, there are many other applications which we will briefly outline below:

There’s more than we can explore on our own! That is why we’re excited to open up these possibilities for the community at large.


Repeatable execution of software is an important capability for debugging, auditability and the other applications described above. Nevertheless, it’s been treated as an afterthought in the technology stacks we all depend on—left as the developer’s responsibility to create “good enough” repeatability for stable tests or a reproducible bug report.

Hermit, as a reproducible container, provides a glimpse of what it would be like if the system stack provided repeatability as an abstraction: a guarantee the developer could rely upon, just like memory isolation or type safety. We’ve only scratched the surface of what is possible with this foundational technology. We hope you’ll check out the open source GitHub repo and help us apply and evolve Hermit.

free widgets for website

Note on prior Art: Earlier academic research on Determinator and dOS represents exploratory work in this area over a decade ago. But these systems were, respectively, a small educational OS and an experimental fork of Linux. Neither was designed as a maintainable mechanism for people to run real software deterministically in practice.

To learn more about Meta Open Source, visit our open source site, subscribe to our YouTube channel, or follow us on Twitter, Facebook and LinkedIn.

First seen at


Now people can share directly to Instagram Reels from some of their favorite apps





More people are creating, sharing and watching Reels than ever before. We’ve seen the creator community dive deeply into video content – and use it to connect with their communities. We’re running a limited alpha test that lets creators share video content directly from select integrated apps to Instagram Reels. Now, creators won’t be interrupted in their workflow, making it easier for them share share and express themselves on Reels.

“With the shift to video happening across almost all online platforms, our innovative tools and services empower creativity and fuel the creator economy and we are proud to be able to offer a powerful editing tool like Videoleap that allows seamless content creation, while partnering with companies like Meta to make sharing content that much easier.”- Zeev Farbman, CEO and co-founder of Lightricks.

Starting this month, creators can share short videos directly to Instagram Reels from some of their favorite apps, including Videoleap, Reface, Smule, VivaVideo, SNOW, B612, VITA and Zoomerang, with more coming soon. These apps and others also allow direct sharing to Facebook , which is available for any business with a registered Facebook App to use.

We hope to expand this test to more partners in 2023. If you’re interested in being a part of that beta program, please fill out this form and we will keep track of your submission. We do not currently have information to share about general availability of this integration.

Learn more here about sharing Stories and Reels to Facebook and Instagram and start building today.

free widgets for website


Q. What is the difference between the Instagram Content Publishing API and Instagram Sharing to Reels?

See also  Facebook buys studio behind collaborative game-making platform 'Crayta'

A: Sharing to Reels is different from the Instagram Content Publishing API, which allows Instagram Business accounts to schedule and publish posts to Instagram from third-party platforms. Sharing to Reels is specifically for mobile apps to display a ‘Share to Reels’ widget. The target audience for the Share to Reels widget is consumers, whereas the Content Publishing API is targeted towards businesses, including third-party publishing platforms such as Hootsuite and Sprout Social that consolidate sharing to social media platforms within their third-party app.

Q: Why is Instagram partnering with other apps?

A: Creators already use a variety of apps to create and edit videos before uploading them to Instagram Reels – now we’re making that experience faster and easier. We are currently doing a small test of an integration with mobile apps that creators know and love, with more coming soon.

Q: How can I share my video from another app to Reels on Instagram?

free widgets for website

A: How it works (Make sure to update the mobile app you’re using to see the new Share to Reels option):

  • Create and edit your video in one of our partner apps
  • Once your video is ready, tap share and then tap the Instagram Reels icon
  • You will enter the Instagram Camera, where you can customize your reel with audio, effects, Voiceover and stickers. Record any additional clips or swipe up to add an additional clip from your camera roll.
  • Tap ‘Next’ to add a caption, hashtag, location, tag others or use the paid partnerships label.
  • Tap ‘Share’. Your reel will be visible where you share reels today, depending on your privacy settings.
See also  A Massive 16000-Member Facebook Group Is Changing How the LA Suburb of Santa Clarita ...

Q: How were partners selected?

A. We are currently working with a small group of developers that focus on video creation and editing as early partners. We’ll continue to expand to apps with other types of creation experiences.

Q: When will other developers be able to access Sharing to Reels on Instagram?

A: We do not currently have a date for general availability, but are planning to expand further in 2023.

Q: Can you share to Facebook Reels from other apps?

free widgets for website

A: Yes, Facebook offers the ability for developers to integrate with Sharing to Reels. For more information on third-party sharing opportunities, check out our entire suite of sharing offerings .

First seen at

Continue Reading


What to know about Presto SQL query engine and PrestoCon





The open source Presto SQL query engine is used by a diverse set of companies to navigate increasingly large data workflows. These companies are using Presto in support of e-commerce, cloud, security and other areas. Not only do many companies use Presto, but individuals from those companies are also active contributors to the Presto open source community.

In support of that community, Presto holds meetups around the world and has an annual conference, PrestoCon, where experts and contributors gather to exchange knowledge. This year’s PrestoCon, hosted by the Linux Foundation, takes place December 7-8 in Mountain View, CA. This blog post will explore some foundational elements of Presto and what to expect at this year’s PrestoCon.

What is Presto?

Presto is a distributed SQL query engine for data platform teams. Presto users can perform interactive queries on data where it lives using ANSI SQL across federated and diverse sources. Query engines allow data scientists and analysts to focus on building dashboards and utilizing BI tools so that data engineers can focus on storage and management, all while communicating through a unified connection layer.

In short, the scientist does not have to consider how or where data is stored, and the engineer does not have to optimize for every use case for the data sources they manage. You can learn more about Presto in a recent ELI5 video below.

Caption: Watch the video by clicking on the image above.

free widgets for website

Presto was developed to solve the problem of petabyte-scale, multi-source data queries taking hours or days to return. These resources and time constraints make real-time analysis impossible. Presto can return results from those same queries in less than a second in most cases, allowing for interactive data exploration.

See also  One FAANG stock is quietly making new highs, and one strategist still sees it as a buy

Not only is it highly scalable, but it’s also extensible, allowing you to build your own connector for any data source Presto does not already support. At a low level, Presto also supports a wide range of file types for query processing. Presto was open sourced by Meta and later donated to the Linux Foundation in September of 2019.

Here are some Presto resources for those who are new to the community:

What is PrestoCon?

PrestoCon is held annually in the Bay Area and hosted by the Linux Foundation. This year, the event takes place December 7-8 at the Computer History Museum. You can register here. Each year at PrestoCon, you can hear about the latest major evolutions of the platform, how different organizations use Presto and what plans the Technical Steering Committee has for Presto in the coming year.

Presto’s scalability is especially apparent as every year we hear from small startups, as well as industry leaders like Meta and Uber, who are using the Presto platform for different use cases, whether those are small or large. If you’re looking to contribute to open source, PrestoCon is a great opportunity for networking as well as hearing the vision that the Technical Steering Committee has for the project in the coming year.

free widgets for website

Explore what’s happening at PrestoCon 2022:

Where is Presto used?

Since its release in November of 2013, Presto has been used as an integral part of big data pipelines within Meta and other massive-scale companies, including Uber and Twitter.

The most common use case is connecting business intelligence tools to vast data sets within an organization. This enables crucial questions to be answered faster and data-driven decision-making can be more efficient.

How does Presto work?

First, a coordinator takes your statement and parses it into a query. The internal planner generates an optimized plan as a series of stages, which are further separated into tasks. Tasks are then assigned to workers to process in parallel.

Workers then use the relevant connector to pull data from the source.

free widgets for website

The output of each task is returned by the workers, until the stage is complete. The stage’s output is returned by the final worker towards the next stage, where another series of tasks must be executed.

The results of stages are combined, eventually returning the final result of the original statement to the coordinator, which then returns to the client.

How do I get involved?

To start using Presto, go to and click Get Started.

We would love for you to join the Presto Slack channel if you have any questions or need help. Visit the community page on the Presto website to see all the ways you can get involved and find other users and developers interested in Presto.

If you would like to contribute, go to the GitHub repository and read over the Contributors’ Guide.

free widgets for website

Where can I learn more?

To learn more about Presto, check out its website for installation guides, user guides, conference talks and samples.

Make sure you check out previous Presto talks, and attend the annual PrestoCon event if you are able to do so.

To learn more about Meta Open Source, visit our open source site, subscribe to our YouTube channel, or follow us on Twitter, Facebook and LinkedIn.

First seen at

free widgets for website
Continue Reading


How to Interpret Webhook Components in the WhatsApp Business Platform





The ways customers want to connect are changing. The WhatsApp Business Platform gives businesses an integrated way to communicate with customers right where they are. In order to integrate properly when using the Cloud API, hosted by Meta, you’ll need to leverage webhooks so applications have a way to respond to events. Webhooks allow your application to monitor three primary events on WhatsApp so you can react with different functionality depending on your goals.

This article looks at these three components, goes through the information they carry, and provides some use-case scenarios to give you an idea of the possibilities.

Interpreting Different Webhook Components

To send and receive messages on WhatsApp, it’s critical to keep track of statuses and errors to help ensure you’re communicating effectively with your customers, which you can do with webhooks.

With webhooks, the WhatsApp Business Platform monitors events and sends notifications when one occurs. These events are one of three components: messages, statuses, and errors.

Let’s explore each of these and examine examples of how you can use them.

free widgets for website


The messages component is the largest of the three event types and contains two core objects:

  • Contacts — which contain information about the message’s sender.

  • Messages — which provide information about a message’s type and contents.

These two event types allow your application to manage and respond to people that interact with your application. The contacts object contains two pieces of information: name and WhatsApp Id. The contact’s name allows your application to use their name without further lookups. In contrast, the contact’s WhatsApp ID lets you keep track of these contacts or use the contacts/ endpoint to add additional functionality.

For instance, you can verify the customer and start the opt-in process within the customer-initiated conversation, which allows you to message them outside the initial 24-hour response window. It’s important to note that only the text, contacts, and location message types provide contact information.

The message object is where the bulk of the information is stored, including the message contents, type of message, and other relevant information. Depending on the message type, the actual payload of the message component can vary widely. It’s crucial to determine the message type to understand the potential payload. Message types include:

  • Text: a standard text-only message

  • Contact: contains a user’s full contact details

  • Location: address, latitude, and longitude

  • Unknown: unsupported messages from users, which usually contain errors.

  • Ephemeral: disappearing messages

  • Media message types: contain information for the specified media file. These types include:

    • Document

    • Image

    • Audio

    • Video

    • Voice

These different data types can have very different uses, from reviewing images and screenshots from concerned customers to collecting information about where to ship goods and send services. To use these different data types most effectively, you can create applications to handle different forms of communication, with functionalities such as:

free widgets for website
  • Ask your customers to provide a shipping or mailing address. You can use the location-based message feature to capture your users’ location to determine where to send their goods and services.

  • Show customers products and communicate product details through a message. You can use the referred_product field within messages to offer your users specific product details. Using this field develops a more personal, conversational shopping experience and customer interactions.

  • Build support functionality that allows customers to take and send images and videos of product concerns, and submit those for a support case. Once the user has submitted a support ticket, the app can track the case — including steps taken towards resolution and conversations between support teams and the customer through WhatsApp — using a unique case identifier.

These are just some potential features you can build using the interactivity provided by webhooks and the message object. These features extend your current communication channels and provide additional options for customers.


Where the messages component provides your application with insight into events that originate directly from your customers, the statuses component keeps track of the results of messages you send and the conversation history. There are six status components:

  • Sent: the application sent your message and is in transit.
  • Delivered: the user’s device successfully received the message.
  • Read: the user has read your message.
  • Deleted: a user deleted a message that you sent.
  • Warning: a message sent by your application contains an item that isn’t available or doesn’t exist.
  • Failed: a message sent by your application failed to arrive.

Status components also contain information on the recipient ID, the conversation, and the pricing related to the current conversation. Conversations on WhatsApp are a grouping of messages within a 24-hour window that are either user-initiated or business-initiated. Keeping track of these conversations is vital, as a new conversation occurs when you send additional responses after the 24-hour period ends.

Some functionality you may want to add to your application based on status events includes:

  • Ensuring your application has sent generated messages, they arrived, and the recipient potentially read them by using a combination of these status types and timestamps within the status object. This information allows your application to follow up with customers if they didn’t engage.
  • Keep analytical information about your application’s messages, especially regarding business-initiated conversations. For example, if your application uses a WhatsApp customer contact list to send offer messages, the status component helps you understand how many were sent, delivered, read, responded to, or failed to measure your campaign’s success.


Finally, the errors component allows your application to receive any out-of-band errors within WhatsApp that affect your platform. These errors don’t stop your application from compiling or working but are typically caused when your application is misusing specific functionality. The following are some typical errors.

Error Code 368, Temporarily Blocked for Policy Violations

If your application violates WhatsApp Business Messaging or Commerce policy, your account may be temporarily banned. You can monitor this and pause your application while troubleshooting.

Error 506, Duplicate Post

If your workflows unintentionally generate duplicate messages, you can monitor this to find the source.

free widgets for website

Error 131043, Message Expired

Sometimes, messages are not sent during their time to live (TTL) duration. Use this code to know which messages to schedule for resending if needed.

Error handling is a broad, complex subject, and there are many other use cases for which you should be implementing error handling. The errors component helps extend your error handling on the WhatsApp Business Platform for greater consistency.


This article took a high-level look at messages, statuses, and errors returned by webhooks and explored ways you can use these three components to expand your application’s functionality.

Messages provide information on customer interactions, statuses give insight into messages your app sends, and error notices enable you to increase your application’s resilience. Webhooks are critical to ensuring your app interacts with customers seamlessly.

The WhatsApp Business Platform’s webhooks provide your applications with real-time data, enabling you to build better experiences as you interact with customers. Ready to know more? Dive deeper into everything the WhatsApp Business Platform has to offer.

free widgets for website

First seen at

See also  NY Times Columnist David Brooks Resigns From Facebook-Funded Nonprofit
Continue Reading