apaam/mlpack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
28,933 Commits 1 Branch 0 Tags
a4ead8bcef5744cc23be86f3d0fe586ebf199ff2
Commit Graph
28933 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Kropp a4ead8bcef author 2023-11-13 14:06:22 -05:00
Adam Kropp 214a731dd7 Merge remote-tracking branch 'origin/adapt_multihead' into adapt_multihead 2023-11-13 14:06:09 -05:00
Adam Kropp d9f9e4f123 Style changes, author 2023-11-13 14:05:47 -05:00
Adam KroppandRyan Curtin 68f15ed77a Update src/mlpack/methods/ann/activation_functions/elish_function.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 14:03:27 -05:00
Adam KroppandRyan Curtin 9451a8fce9 Update src/mlpack/tests/ann/activation_functions_test.cpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 14:02:48 -05:00
Adam KroppandRyan Curtin 008aabb8a0 Update src/mlpack/tests/ann/layer/multihead_attention.cpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 13:50:42 -05:00
Adam KroppandRyan Curtin fb94b6d174 Update src/mlpack/tests/ann/activation_functions_test.cpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 13:50:30 -05:00
Adam KroppandRyan Curtin 89f93d1272 Update src/mlpack/tests/ann/layer/layer_norm.cpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 13:50:14 -05:00
Adam KroppandRyan Curtin 5f385ff946 Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 13:49:58 -05:00
Adam KroppandRyan Curtin eb3e7f2a48 Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-13 13:47:17 -05:00
Adam Kropp f5be365000 Correct calculation of srcSeqLen with more than 2 input dimensions 2023-11-13 07:55:46 -05:00
Adam Kropp c990b93917 Comment unused parameter 2023-11-10 16:12:34 -05:00
Adam Kropp 39f7a10fdc Comment out unused params 2023-11-10 15:50:28 -05:00
Adam Kropp 2bdae7d438 Add MultiheadAttention test, fix dimension calculation 
Also added some language to the comments about the sequence lengths.
 2023-11-10 14:27:00 -05:00
Adam Kropp 3363707881 Add in LayerNorm Test 2023-11-10 08:29:13 -05:00
Adam Kropp 7b5e4d267b Merge remote-tracking branch 'upstream/master' into adapt_multihead 2023-11-09 16:01:28 -05:00
Adam Kropp 65b4878ad0 Clairfy input dimensions documentation, check valid input dims 2023-11-08 09:18:21 -05:00
Adam KroppandRyan Curtin d91c5f0f31 Update src/mlpack/methods/ann/layer/c_relu_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-08 08:14:11 -05:00
Adam KroppandRyan Curtin f6d78abf42 Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-08 08:13:05 -05:00
Adam KroppandRyan Curtin 1b8313eafb Update src/mlpack/methods/ann/layer/flexible_relu_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-08 08:05:27 -05:00
Adam KroppandRyan Curtin c40787c3f8 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-08 08:05:09 -05:00
Adam KroppandRyan Curtin ef3183ee68 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-08 08:04:56 -05:00
Adam KroppandRyan Curtin 517bb0b27f Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-08 07:55:13 -05:00
Adam Kropp cc99b4ac90 Update activation functions to take input and output values 
Updated test cases as well.
 2023-11-08 00:13:57 -05:00
Ryan Curtin 7af47e22bd Fix numClasses member in SoftmaxRegression::Train() (#3553) 2023-11-05 14:32:01 +01:00
Adam Kropp 45968109ac Add LayerNorm and MultiAttentionHead tests 2023-11-03 12:14:19 -04:00
Adam Kropp 360c17c7e5 Fix c_relu test and backward method 
Defined the derivative at 0 to be exactly 0 (rather than -1 or 1).  Fixed the expected derivatives in the test case.
 2023-11-03 11:07:04 -04:00
Adam Kropp b7ab3ba063 Fix ftswish to use input instead of output 
I corrected the test case too (calculating the values separately).  It appears that they had been calculated based on the ouputs, which from my reading of the paper is incorrect.
 2023-11-03 08:34:17 -04:00
Adam Kropp 20a48b8c72 Unnecessary alias 2023-11-02 17:45:16 -04:00
Adam Kropp a5537825c9 Optimization of Backward() 
also replaced DBL_MAX with a type-safe alternative.
 2023-11-02 17:15:09 -04:00
Adam Kropp ce846ea032 Defining Backward in terms of the inputs 
I believe the backward function should calculate the gradient in terms of the inputs, not the outputs.
 2023-11-02 17:06:44 -04:00
Adam Kropp 8401f64fd9 Doc updates 2023-11-02 16:36:42 -04:00
Adam Kropp 8cec72dac8 (correctly) change Backward to use input instead of output 2023-11-02 16:03:05 -04:00
Adam Kropp 636c4fc29e consolidate constructors with default arg 2023-11-02 15:37:00 -04:00
Adam Kropp b0a60bdd85 Remove dependency on input in Backward call 2023-11-02 15:35:11 -04:00
Adam Kropp 8dce94d42b Update multihead_attention_impl.hpp 
Change the attnMask and keyPaddingMask to be 0 or 1 instead of -inf or 0
 2023-11-02 15:33:00 -04:00
Adam Kropp e24bfd20ab Remove input sizing (srcSeqLen, embedDim) from constructor. 
Add doc for attnMask, keyPaddingMask
Remove attnMask, keyPaddingMask from constructor.
Serialize selfAttention
Remove softMaxOutput temp member var.
 2023-11-02 12:29:55 -04:00
Adam Kropp 45f88d1526 Clean-up to fix compilation/test. 2023-11-02 12:27:49 -04:00
Adam KroppandRyan Curtin b6a2c58ba7 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 10:56:37 -04:00
Adam KroppandRyan Curtin d9e7d612c4 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 10:02:04 -04:00
Adam KroppandRyan Curtin c482407fb2 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 10:01:22 -04:00
Adam KroppandRyan Curtin 21fe33e1f2 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 09:48:17 -04:00
Adam KroppandRyan Curtin 6dd71c19a7 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 09:45:22 -04:00
Adam KroppandRyan Curtin 346ea05d3e Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 09:44:15 -04:00
Adam KroppandRyan Curtin b79f69082c Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 09:42:59 -04:00
Adam KroppandRyan Curtin 1907e527b4 Update src/mlpack/methods/ann/layer/multihead_attention_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-02 09:41:40 -04:00
Adam KroppandRyan Curtin 9388c6f698 Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-01 20:26:50 -04:00
Adam KroppandRyan Curtin bec84f4cab Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-01 20:26:16 -04:00
Adam KroppandRyan Curtin 7d84005e52 Update src/mlpack/methods/ann/layer/multihead_attention.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-01 20:25:07 -04:00
Adam KroppandRyan Curtin 4e282dd342 Update src/mlpack/methods/ann/layer/layer_norm_impl.hpp 
Co-authored-by: Ryan Curtin <ryan@ratml.org>
 2023-11-01 20:24:15 -04:00