On Staying Confident

Last April, I submitted the manuscript of Zeus to the Spring deadline of NSDI. I was fairly confident about my research capability, since I managed to submit something that looks like a good paper by a very stringent deadline.

After a good three-week break, I came back to research in late May, exploring potential future directions. I suggested some ideas and directions during meetings with my advisor, and basically everything was killed. They were not just killed, but killed with very good reasons that seemed obvious in retrospect. Quickly in several weeks, I started to feel very unconfident to the point where I was afraid to talk with my advisor, although I knew logically that killing bad ideas in early stages is only saving me time.

Then suddenly in mid-July, we were notified that Zeus was accepted to appear at NSDI. My mood swung to the other end of confidence, especially because this was our first submission of the paper to any conference. I began to prepare the camera-ready version, together with Zeus’s open source repository and homepage. Finding new research directions was paused for a week or two, since we wanted Zeus to be posted on arXiv as soon as possible and have it collect citations from early on.

While polishing Zeus, I was thinking about what could be a follow-up work of it. Then I came up with another idea, an idea I liked no less than all the dead ideas in June, and presented that to my advisor. And he said it was very good. Now I’m working on that idea, and just found last week that my hypothesis was true at least for a limited set of workloads, and the potential gains can be quite large.

Looking back starting from April, when I submitted Zeus to NSDI, me as a researcher did not change that much. Most of the time I was on vacation, or was pouring time into polishing Zeus. However, my level of confidence in terms of research capability fluctuated greatly, which didn’t make any sense. If me in April and me today are similar researchers, there is no reason to be sometimes confident and sometimes not. Rather, my confidence level should be determined by my best times, which is currently mid-July, when Zeus was accepted. External factors, for example whether my ideas are well accepted by people, do not change my potential to do great work.