tag:blogger.com,1999:blog-4446292666398344382.post3407292445863767621..comments2020-10-14T21:49:04.423-07:00Comments on Machined Learnings: Constrained Best M: Reduction to Constrained CSMCPaul Mineirohttp://www.blogger.com/profile/05439062526157173163noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-4446292666398344382.post-25075694313697874122010-12-31T13:36:04.322-08:002010-12-31T13:36:04.322-08:00Reductions are about relating some problem that is...Reductions are about relating some problem that is less understood (e.g., CSMC) to some problem that is better understood (e.g., binary classification or regression). It's true that if we reduce to a subproblem and then do badly on the subproblem, we will do badly on the original problem. However the idea is we feel like we have a lot of experience on the basic subproblems like binary classification so we expect to do rather well.<br /><br />Worst case analysis can be deceiving sometimes, but for CSMC in particular on difficult (noisy) data sets, empirical performance does appear to follow the worst-case bounds.<br /><br />Re: SVM_Perf or SVM_MAP. Reductions are about solving a problem by converting into a problem with an already existing solution. Another approach is to directly solve the original problem. SVM_Perf takes the direct approach for ranking losses such as AUC and precision at k. SVM_MAP takes the direct approach for the ranking loss mean average precision.<br /><br />Ranking losses (AUC, prec@k, MAP) all assume there are "good results" and "bad results" and define different ways of assigning a value to a large bit string (the best bit string being 00000....01...11111) which represents the ranking. CSBM is different because it is assumed that the decisions have scalar values associated with them and wants to maximize the total summed value of the set. In advertising, in particular, there are different amounts of money made when different ads are shown, so paying attention to that might improve performance in practice.<br /><br />In the case of a reduction approach to a problem and a direct approach to a problem both of which are operating on the same loss function, I would recommend empirical evaluation, especially in the case where somebody like Jochaims is providing high quality software than you can download and test drive. <br /><br />When are attacking a problem that hasn't gotten attention in the literature, I would suggest looking for a reduction rather than a direct algorithm, as the former often seems easier to come up with and easier to prove that it does something reasonable; as a bonus, you get to reuse great software for the reduced learning piece.Paul Mineirohttps://www.blogger.com/profile/05439062526157173163noreply@blogger.comtag:blogger.com,1999:blog-4446292666398344382.post-33800750349937562752010-12-30T22:33:58.663-08:002010-12-30T22:33:58.663-08:00Since I am still new to reduction method, so proba...Since I am still new to reduction method, so probably my questions are irrelevant.<br /><br />1. I find all the papers about reduction method try to guarantee the regret bound or error bound, but is it the only thing important? Consider the simplest example to solve binary classification problem with regression method, with the regret bound and error bound proved, if the regression method work well on the transformed dataset, then the final result of the reduction method should also be well. But shouldn't we prove that the regression method can work well on the transformed dataset indeed?<br /><br />2. I am praticularly interested in the topics about CSMC and CSBM since they are directly related to my current project. I think the reduction of CSMC and CSBM to regression is quite similar to the work of Thorsten Joachims (SVM_Perf and SVM_MAP respectively, some extensions are needed, of course), so how should they be compared?Anonymoushttps://www.blogger.com/profile/17392053840179839335noreply@blogger.com