Everything Is a Leaky Abstraction*

*in ma­chine learn­ing

One of my fa­vorite blog posts of all time was writ­ten by Andrej Karpathy nearly ten years ago. In it, he de­tails how he would re­quire the stu­dents of his CS231n Deep Learning class at Stanford to im­ple­ment the for­ward and back­ward passes of neural net­works by hand (that is, in Python with NumPy). Why? Because he wanted his stu­dents to leave his class with a vis­ceral un­der­stand­ing of back­prop­a­ga­tion. They should know how it works, why it works, and thus how to fix it when it stops work­ing.

It’s a great read, and I highly rec­om­mend that any­one and every­one in­ter­ested in ML pe­ruse it. You can find it here.

Some Sealed Abstractions

There are some things that are sealed ab­strac­tions (that is, they are not leaky). But I do not think any of them ex­ist in the se­ri­ous study of ma­chine learn­ing. Some ex­am­ples in­clude:

• Lossless com­pres­sion al­go­rithms.
• TCP/IP
• TLS
• PostgreSQL

Sure, with some of these things you oc­ca­sion­ally need to un­der­stand the un­der­ly­ing sys­tem for per­for­mance tun­ing. But knowl­edge of the un­der­ly­ing sys­tem is rarely the dif­fer­ence be­tween suc­cess and fail­ure.

ML Is Not like Normal Engineering

ML is not that way. To suc­cess­fully train a model, you need to have a clear un­der­stand­ing of every as­pect of the sys­tem, all the way down to how in­di­vid­ual weights and bi­ases are up­dated. It is not a kind dis­ci­pline.

Obviously, there is Karpathy’s ex­am­ple of back­prop­a­ga­tion, but it goes fur­ther than that. Every as­pect of model ar­chi­tec­ture can in­flu­ence the shape of the cost func­tion op­ti­mized via back­prop­a­ga­tion, which means every as­pect of model ar­chi­tec­ture is leaky.

I have got the sense that lay­men be­lieve that any prob­lem can be solved by throw­ing a trans­former at it. If only it were so sim­ple.

Successful use of a trans­former (or any model ar­chi­tec­ture, for that mat­ter) de­pends on a good un­der­stand­ing of how it works and how it af­fects the cost func­tion.

Take Your Time

As with any kind of sys­tem with a Rube Goldberg–level com­plex­ity, it is best to take your time. Go slow. Skip that ex­tra pot of cof­fee. It’s bet­ter to be right slowly than wrong very fast. Un­der­stand every as­pect of your model and train­ing reg­i­men be­fore you start, not af­ter.