Lda v1 #311
Open
Lda v1 #311
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This work is based on the original work by Xiaocheng Tang <[email protected]> in madlib#75. This PR adds two main features: - A Minibatch solver that takes as input a batch of data - SVM code that takes advantage of the minibatch Closes madlib#229 Co-authored-by: Nikhil Kak <[email protected]> Co-authored-by: Xiaocheng Tang <[email protected]>
JIRA: MADLIB-1201 Fixed the issue of output of lda_train and lda_get_word_topic_count not matching each other. Added test case in install check. See jira for more details and example. Also added a install check that validates that the output of lda_train and lda_get_word_topic_count are consistent with each other. See jira for more details and example.
JIRA: MADLIB-1160 This commit adds a helper function, which will map each wordid with corresponding topicid that get assigned in output table. Duplicate lines are removed from the final result. Also adds a workaround for GPDB4.3 svec In GPDB4.3, we cannot call madlib.svec directly on a text format.Instead, we have to call madlib.svec_from_string to convert the text. This commit fix this issue so the new helper function madlib.lda_get_word_topic_mapping can work on both gpdb5 and gpdb4.
JIRA:MADLIB-1160 This commit fixes the topicid inconsistency in madlib.lda_train and madlib.lda_get_topic_desc, where the former one uses 0 based index and the latter uses 1 index. Now they will all start at 0.
JIRA: MADLIB-1160 Previously, madlib.lda_get_topic_desc got top k - 1 words in the result table. This commit fixed it to be top k.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User doc updates for LDA and term frequency.
At the same time, I made minor updates to user docs for PageRank, train-test split and matrix ops and combined into 1 PR.