Mostly from tweets of Vijay Chidambaram
General
- The most important aspect is problem selection. The problem has to be timely and significant. By reading a lot of SOSP/OSDI papers, you develop a sense of whether a given problem is significant enough. If you tackle the “wrong” problem, paper won’t get accepted no matter what.
- Building a real system (to the point it can run actual user workloads). It is rare to have SOSP or OSDI papers based on simulation. Measurement papers don’t build a real system, but then have the burden of interesting, surprising insights.
- Most SOSP/OSDI papers have a strong evaluation.
- SOSP and OSDI papers usually have impressive results (though not always). If the focus is perf, perf is improved by a lot. If the focus is finding bugs, a lot of new bugs are discovered. If the focus is measurement, the paper usually contains multiple interesting insights.
- Writing is super super important. It doesn’t matter how good your results are if nobody can understand them. The problem must be clear and well-motivated, the insights behind the solution described, and results must be explained. Writing is the most common failure mode. Rejected papers at SOSP and OSDI usually contain amazing results which are not explained well.
- A good SOSP/OSDI weaves a tightly-knot, engrossing story:
Here’s this important problem. But look, current solutions are bad! Insight: we can solve it by doing X! Here’s how we built a system around this insight. Since we do X, perf is up Y%! We can do new thing Z!
- The “story” aspect is the toughest to obtain, because a bunch of incoherent techniques won’t cut it. It has to make sense to the reviewer intuitively. The story aspect improves (for me) as we rewrite the paper – the story in the first draft is rarely what gets submitted.
Evaluation
Needs to answer three questions:
- What are the strengths and weakness of your system? –> Microbenchmarks
- How does it compare to the state-of-the-art approaches? –> Comparison with open source systems
- How does it improve the performance of real applications? -> Need to find real applications that can run on top of the system
-
Other conferences don’t place a lot of value on #3, but SOSP/OSDI always do this. Its one of the stand-out features of the conference. So if you are hoping to have your paper appear there, you have to put in the work to do this.
-
There is a good reason SOSP/OSDI insist on performance impact on real applications. Many research projects optimize for use-cases that are not common at all. It is troubling if there isn’t a single application for the use case.
-
Of course, all evaluation isn’t about performance. If your approach enables new functionality, then you have to show why its hard/inefficient to do it with current approaches.
-
Whatever the idea is, in the end, OSDI likes to see a real system that can be evaluated to get quantitive numbers. Earlier-stage projects are not a good fit I guess.
-
If the system allows you to perform actions that weren’t possible before, showing that is a good evaluation too. Authors usually show it is much harder and more inefficient to do the same thing in an existing system.