{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch

Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|

Performer Pytorch | 777 | 4 | 2 years ago | 77 | February 02, 2022 | 32 | mit | Python | ||

An implementation of Performer, a linear attention-based transformer, in Pytorch | ||||||||||

Pytorch_block_sparse | 379 | 3 years ago | 9 | other | C++ | |||||

Fast Block Sparse Matrices for Pytorch | ||||||||||

Sequitur | 293 | a year ago | 9 | January 28, 2021 | 6 | mit | Python | |||

Library of autoencoders for sequential data | ||||||||||

Nngeometry | 155 | 2 months ago | 1 | February 24, 2021 | 2 | mit | Python | |||

{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch | ||||||||||

Butterfly | 76 | 2 years ago | 12 | apache-2.0 | Python | |||||

Butterfly matrix multiplication in PyTorch | ||||||||||

Structured Nets | 48 | a year ago | 6 | apache-2.0 | Python | |||||

Structured matrices for compressing neural networks | ||||||||||

Lssvm | 33 | 4 months ago | 3 | mit | Python | |||||

Python implementation of Least Squares Support Vector Machine for classification on CPU (NumPy) and GPU (PyTorch). | ||||||||||

Openai Gemm.pytorch | 21 | 7 years ago | Python | |||||||

PyTorch bindings for openai-gemm | ||||||||||

Pytorch_sym3eig | 18 | a year ago | 1 | apache-2.0 | C++ | |||||

Pytorch extension: Batch-wise eigencomputation for symmetric 3x3 matrices | ||||||||||

Mogrifier Lstm Pytorch | 10 | 3 years ago | Python | |||||||

Implementation of Mogrifier LSTM in PyTorch |

Alternatives To NngeometrySelect To Compare

Alternative Project Comparisons

Readme

NNGeometry allows you to:

- compute
**Fisher Information Matrices**(FIM) or derivates, using efficient approximations such as low-rank matrices, KFAC, diagonal and so on. - compute finite-width
**Neural Tangent Kernels**(Gram matrices), even for multiple output functions. - compute
**per-examples jacobians**of the loss w.r.t network parameters, or of any function such as the network's output. - easily and efficiently compute linear algebra operations involving these matrices
**regardless of their approximation**. - compute
**implicit**operations on these matrices, that do not require explicitely storing large matrices that would not fit in memory.

In the Elastic Weight Consolidation continual learning technique, you want to compute . It can be achieved with a diagonal approximation for the FIM using:

```
F = FIM(model=model,
loader=loader,
representation=PMatDiag,
n_output=10)
regularizer = F.vTMv(w - w_a)
```

If diagonal is not sufficiently accurate then you could instead choose a KFAC approximation, by just changing `PMatDiag`

to `PMatKFAC`

in the above. Note that it internally involves very different operations, depending on the chosen representation (e.g. KFAC, EKFAC, ...).

You can visit the documentation at https://nngeometry.readthedocs.io.

More example usage are available in the repository tfjgeorge/nngeometry-examples.

We welcome any feature request or bug report in the issue tracker.

We also welcome contributions, please submit your PRs!

If you use NNGeometry in a published project, please cite our work using the following bibtex entry

```
@software{george_nngeometry,
author = {Thomas George},
title = {{NNGeometry: Easy and Fast Fisher Information
Matrices and Neural Tangent Kernels in PyTorch}},
month = feb,
year = 2021,
publisher = {Zenodo},
version = {v0.2.1},
doi = {10.5281/zenodo.4532597},
url = {https://doi.org/10.5281/zenodo.4532597}
}
```

This project is distributed under the MIT license (see LICENSE file). This project also includes code licensed under the BSD 3 clause as it borrows some code from owkin/grad-cnns.

Popular Pytorch Projects

Popular Matrices Projects

Popular Machine Learning Categories

Related Searches

Get A Weekly Email With Trending Projects For These Categories

No Spam. Unsubscribe easily at any time.

Python

Pytorch

Kernel

Neural

Matrices