The CS Publication Problem
There is a problem in a lot of computer science, and especially in machine learning, and it is the problem of lore. In any field moving quickly, new ideas, new methods, and new results will come so quickly that they will outstrip the machinery of scientific dissemination and scientific review and this introduces an uncomfortable struggle for balance between moving forward and building mystical castles in the air.
Let me backtrack a little: I think much of Computer Science, as a field, has really just given up on being an empirical science or a legitimate academic discipline. Professors are apparently getting tenure based mostly upon their "reviewed" conference publications, and while I just recently got two of those published -- and will play them up for all they are worth -- I have to acknowledge that what passes for review at CS conferences is not what I would call review coming as I do from other disciplines.
Both of my recent papers in CS were reviewed in the sense that an unnamed panel of 3 or 4 conference organizers read the papers, and made a lot of snarky comments. And that was it. I was told to revise the manuscript "taking the reviewers comments into account" but without any specific demand to do so and after having already been told the paper was accepted. There was no letter written to respond to the reviewers, no second pass to check my work, really none of the apparatus I would call scientific publication review. In short, some suggestions were made, I ignored or adopted the changes, the manuscript was resubmitted, and then the wheels of publication rolled on.
I'm sure my CS colleagues will tell me this is necessary in their undoubtedly special discipline which moves, lightning fast, compared to classical academic disciplines like math, stats, and cognitive science (to speak to my own background), or even disciplines like physics (which is having something similar happen with the wide-spread adoption of the practice of just using pre-prints over reviewed articles. (Winter is coming, physics, winter is coming. And CS is pretty icy already.)
But I suspect it is just systemic laziness and the ever-present push of capitalism. When your pay is based on the number of published papers, it becomes very easy to accept grading those publications on a curve, espically when you giving inflated grades is met with others doing the same for you.
No doubt this affects all of academic writing and no doubt it is getting worse, but the real question is what can we do about it? Or do we all just become little isolated islands defined by our own internal expert opinions and beliefs?
Ok, no time left for lore...I will have to come back to that later.