Machine Learning Meetup
This weekly seminar series, hosted by the computer vision research team at FeatureX, is open to students and professionals who share an interest in gathering with like-minded machine learning researchers. This series focuses on current and influential papers in machine learning, and brings active participants together around one relevant paper each week. The presenter will introduce the background of the paper and review the findings. Attendees are expected to have read the paper and be ready to participate in group discussions about the research content and its implications.
Space is limited and RSVP’s are mandatory for this event. Please email Emily Rogers at email@example.com if you plan to attend. If your plans change, please update us so we can offer space to someone else.
Paper Title and Link: Measuring the Intrinsic Dimension of Objective Landscapes by Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski
Abstract: Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.