Maximum entropy models

For more info about our services contact : help@bestpfe.com

Table of contents

Détail des contributions 
0.1 Modélisation de la langue naturelle
0.2 Méthodes existantes
0.2.1 Méthodes par lissage
0.2.2 Entropie maximale et maximum de vraisemblance
0.2.3 Méthodes récentes
0.3 Motivations
0.4 Contributions et plan de la thèse
0.4.1 Pénalités structurées pour la modélisation du langage
0.4.2 Apprentissage de la taxonomie
0.4.3 Plan de la thèse
1 Introduction 
1.1 Problem of modeling language
1.2 Traditional approaches
1.2.1 Smoothing models
1.2.2 Maximum entropy models
1.2.3 Predictive performance and computational efficiency
1.3 Recent trends
1.3.1 Bayesian models
1.3.2 Distributed representations
1.4 Summary and motivation
1.5 Contributions and outline
2 Learning with structured penalties 
2.1 Principle of empirical risk minimization
2.2 Penalized loss and structural risk minimization
2.2.1 Unstructured penalties
2.2.2 Structured penalties
2.3 Optimizing penalized loss
2.4 Proximal minimization algorithms
2.4.1 Proximal operators for penalties
2.5 Conclusion
3 Log-linear language model 
3.1 Introduction to Generalized Linear Models
3.2 Log-linear language model
3.3 Suffix encoding in tries and trees
3.3.1 Suffix trie structured vectors
3.3.2 Word-specific suffix trie structured vectors
3.3.3 Constraining to positive orthant
3.3.4 Word-specific suffix tree-structured vectors
3.3.5 Complexity improvements from trees
3.4 Models with unstructured penalties
3.4.1 Proximal projection with unstructured penalties
3.4.2 Performance evaluation with unstructured penalties
3.5 Models with structured penalties
3.5.1 Proximal projection with `T 2 -norm
3.5.2 Proximal projection with `T 1 -norm
3.5.3 Performance evaluation with structured norms
3.5.4 Feature weighting to avoid overfitting
3.5.5 Analysis of perplexity improvements
3.6 Time complexity of proximal operators
3.6.1 Properties of `T 1 -norm
3.6.2 Fast proximal projection with `T 1 -norm
3.7 Conclusion
4 Efficient taxonomy learning 
4.1 Normalization in multi-class log-linear models
4.2 Tree embedding of classes
4.2.1 Measuring the class proximity
4.2.2 Split and merge operations
4.2.3 Top-down procedure
4.2.4 Bottom-up procedure
4.3 Node-specific classifiers
4.4 Evaluation of taxonomy methods
4.5 Conclusion
5 Concluding remarks 
Appendices 
A Smoothing and maximum entropy models
A.1 Smoothing models
A.1.1 Redistribute mass to handle unseen sequences
A.1.2 Avoid overfitting using lower order distributions
A.2 Iterative scaling methods
A.2.1 Generalized Iterative Scaling
A.2.2 Improved Iterative Scaling
A.2.3 Sequential Conditional Generalized Iterative Scaling
B Synthetic Data Generation
C Effect of parameters